Re: Recent NTP pool traffic increase

[Harlan Stenn]

On 12/30/16 11:26 AM, Emille Blanc wrote:
> Ah, but who do you trust? Trump, Putin, or Xi's clock?
> 
> That said, we use a Stratum2 clock for our AS, which syncs using GPS
> at $dayjob. So... I guess we trust Trump's clock.
> 
> Perhaps there's a market for a device that takes GPS, GLONASS, and
> Beidou, and references the three for sanity checks in the event of
> $unforseen_circumstance. Assuming such a thing were possible -
> admittedly I know little about GLONASS, and even less about Beidou.

Why are you trying to re-invent a properly configured S2 NTP server?

H
--
> "Perhaps some kind of death clock?"
> 
> -Original Message- From: NANOG
> [mailto:nanog-bounces+emille=abccommunications@nanog.org] On
> Behalf Of Allan Liska Sent: December-30-16 11:09 AM To: Majdi S.
> Abbas; Laurent Dumont Cc: nanog@nanog.org Subject: Re: Recent NTP
> pool traffic increase
> 
> 
> On 12/30/2016 at 1:20 PM, "Majdi S. Abbas"  wrote:On Thu, Dec 22,
> 2016 at 11:31:08PM -0500, Laurent Dumont wrote:
>> What I mostly meant is that there should be a regulated,
> industry-wide
>> effort in order to provide a stable and active pool program. With
> the
>> current models, a protocol that is widely used by commercial
>> devices
> is
>> being supported by the time and effort of volunteers around the
> world.
> 
> Who's authoritative for time? Even the national labs aren't -- UTC is
> figured well after the fact.
> 
> In the United States that would the United States Naval Observatory 
> (USNO) Master Clock (http://tycho.usno.navy.mil/).  You can read
> more about it here: 
> http://motherboard.vice.com/read/demetrios-matsakis-and-the-master-clock
>
>  allan
> 
> 

-- 
Harlan Stenn <st...@nwtime.org>
http://networktimefoundation.org - be a member!

Re: Recent NTP pool traffic increase

[Majdi S. Abbas]

On Fri, Dec 30, 2016 at 02:08:50PM -0500, Allan Liska wrote:
> In the United States that would the United States Naval Observatory
> (USNO) Master Clock (http://tycho.usno.navy.mil/).  You can read more
> about it here:
> http://motherboard.vice.com/read/demetrios-matsakis-and-the-master-clock

One of the things I have learned as a time hobbyist is that if
something involves time, and you think there is a simple answer, you
are probably wrong.  :)

USNO is our military time keeper -- NIST keeps time for civil
purposes, and while they coordinate to stay in reasonably close
proximity, even they don't agree.  Even better, the GPS clocks are
run by (and corrections distributed) by the Air Force, not the Navy.
And they have made mistakes in recent memory.

From an international perspective, BIPM is responsible for UTC,
but it is only figured well after the fact.  We distribute "UTC" via
NTP, but it's not true UTC since that is not figured in real time,
it's much, much coarser, and everyone's local views differ anyway.

For an idea just how many components there are, take a look
at BIPM's Circular T:

ftp://ftp2.bipm.org/pub/tai//Circular-T/cirthtm/cirt.347.html

But back to the point...while UTC is an international
time scale, individual national labs and institutions keep their
own views of it, and correct periodically...then they distribute
these timescales, and in some fashion we attempt to get a coarse
version of it onto the Internet in real time.

There is no one authority responsible for this, and you
may take time from any one (or more) of them that you choose.
And for this reason, there is no single authority for time 
distribution on the Internet -- because there is none for the
world as a whole, either.

We /can/ have an authoritative system for something 
like host naming, where it's comparatively easy to produce a
single authoritative source.  Timing is not nearly such a
simple subject.

Cheers,

--msa

RE: Recent NTP pool traffic increase

[Jeff McAdams]

On Fri, December 30, 2016 14:26, Emille Blanc wrote:
> Ah, but who do you trust? Trump, Putin, or Xi's clock?
>
>
> That said, we use a Stratum2 clock for our AS, which syncs using GPS at
> $dayjob. So... I guess we trust Trump's clock.
>
>
> Perhaps there's a market for a device that takes GPS, GLONASS, and
> Beidou, and references the three for sanity checks in the event of
> $unforseen_circumstance. Assuming such a thing were possible - admittedly
> I know little about GLONASS, and even less about Beidou.

I was casually (yes, really) looking around at some GNSS modules available
the other day, and readily found quite affordable ones that can receive at
least 2 at a time, of those systems.  GPS and GLONASS being the most
complete, so probably the best bang-for-the-buck.

I'm only an armchair time-nut, so I can't really speak to their precision
and such, so this is very much a FWIW post.

>
> -Original Message-
> From: NANOG [mailto:nanog-bounces+emille=abccommunications@nanog.org]
> On Behalf Of Allan Liska
> Sent: December-30-16 11:09 AM
> To: Majdi S. Abbas; Laurent Dumont
> Cc: nanog@nanog.org
> Subject: Re: Recent NTP pool traffic increase
>
>
>
> On 12/30/2016 at 1:20 PM, "Majdi S. Abbas"  wrote:On Thu, Dec 22, 2016
> at 11:31:08PM -0500, Laurent Dumont wrote:
>> What I mostly meant is that there should be a regulated,
>>
> industry-wide
>> effort in order to provide a stable and active pool program. With
> the
>> current models, a protocol that is widely used by commercial devices
> is
>> being supported by the time and effort of volunteers around the
> world.
>
> Who's authoritative for time? Even the national labs aren't --
> UTC is figured well after the fact.
>
>
> In the United States that would the United States Naval Observatory
> (USNO) Master Clock (http://tycho.usno.navy.mil/).  You can read more
> about it here:
> http://motherboard.vice.com/read/demetrios-matsakis-and-the-master-clock
>
>
> allan
>
>
>


-- 
Jeff

RE: Recent NTP pool traffic increase

[Allan Liska]

On 12/30/2016 at 2:26 PM, "Emille Blanc"  wrote:Ah, but who do you
trust? Trump, Putin, or Xi's clock?

That said, we use a Stratum2 clock for our AS, which syncs using GPS
at $dayjob. So... I guess we trust Trump's clock.

Perhaps there's a market for a device that takes GPS, GLONASS, and
Beidou, and references the three for sanity checks in the event of
$unforseen_circumstance. Assuming such a thing were possible -
admittedly I know little about GLONASS, and even less about Beidou.

"Perhaps some kind of death clock?"
Personally, when it goes online, I plan on trusting the "Jeff Bezos"
clock :)
http://www.1yearclock.net/
allan

RE: Recent NTP pool traffic increase

[Emille Blanc]

Ah, but who do you trust? Trump, Putin, or Xi's clock?

That said, we use a Stratum2 clock for our AS, which syncs using GPS at 
$dayjob. So... I guess we trust Trump's clock.

Perhaps there's a market for a device that takes GPS, GLONASS, and Beidou, and 
references the three for sanity checks in the event of $unforseen_circumstance. 
Assuming such a thing were possible - admittedly I know little about GLONASS, 
and even less about Beidou.

"Perhaps some kind of death clock?"

-Original Message-
From: NANOG [mailto:nanog-bounces+emille=abccommunications@nanog.org] On 
Behalf Of Allan Liska
Sent: December-30-16 11:09 AM
To: Majdi S. Abbas; Laurent Dumont
Cc: nanog@nanog.org
Subject: Re: Recent NTP pool traffic increase


On 12/30/2016 at 1:20 PM, "Majdi S. Abbas"  wrote:On Thu, Dec 22, 2016
at 11:31:08PM -0500, Laurent Dumont wrote:
> What I mostly meant is that there should be a regulated,
industry-wide
> effort in order to provide a stable and active pool program. With
the
> current models, a protocol that is widely used by commercial devices
is
> being supported by the time and effort of volunteers around the
world.

Who's authoritative for time? Even the national labs aren't --
UTC is figured well after the fact. 

In the United States that would the United States Naval Observatory
(USNO) Master Clock (http://tycho.usno.navy.mil/).  You can read more
about it here:
http://motherboard.vice.com/read/demetrios-matsakis-and-the-master-clock

allan

Re: Recent NTP pool traffic increase

[Allan Liska]

On 12/30/2016 at 1:20 PM, "Majdi S. Abbas"  wrote:On Thu, Dec 22, 2016
at 11:31:08PM -0500, Laurent Dumont wrote:
> What I mostly meant is that there should be a regulated,
industry-wide
> effort in order to provide a stable and active pool program. With
the
> current models, a protocol that is widely used by commercial devices
is
> being supported by the time and effort of volunteers around the
world.

Who's authoritative for time? Even the national labs aren't --
UTC is figured well after the fact. 

In the United States that would the United States Naval Observatory
(USNO) Master Clock (http://tycho.usno.navy.mil/).  You can read more
about it here:
http://motherboard.vice.com/read/demetrios-matsakis-and-the-master-clock

allan

Re: Recent NTP pool traffic increase

[Majdi S. Abbas]

On Thu, Dec 22, 2016 at 11:31:08PM -0500, Laurent Dumont wrote:
> What I mostly meant is that there should be a regulated, industry-wide
> effort in order to provide a stable and active pool program. With the
> current models, a protocol that is widely used by commercial devices is
> being supported by the time and effort of volunteers around the world.

Who's authoritative for time?  Even the national labs aren't --
UTC is figured well after the fact.  

Thing about the pool is -- you may use it, you don't have to.
You're welcome to provide your own services, including to your
customers.

There has to be one sole DNS -- there isn't one sole source of
time.

--msa

Re: Recent NTP pool traffic increase

[Rich Kulawiec]

On Tue, Dec 20, 2016 at 10:27:18PM -0500, Laurent Dumont wrote:
> To be honest, the fact that NTP is still something managed by volunteers and
> not a regulated entity (a bit like DNS) is mind boggling.

I don't see why.  Look back on 30-ish years of domain registration, I think
that it was far more responsibly, ethically, and professionally managed
when it was handled by volunteers.

---rsk

Re: Recent NTP pool traffic increase (update)

[Harlan Stenn]

Hi Fujimura-san,

On 12/24/16 6:11 AM, FUJIMURA Sho wrote:
> Hello,
> 
> I know 133.100.9.2 and 133.100.11.8 are listed.
> The Server Contact is old information.
> So, I sent e-mail to webmas...@ntp.org a few times.
> But, I have't received e-mail from them.
> I'd like them to change the information.
> Is there the person knowing the contact information to ntp.org?

I don't recall seeing the emails you sent to webmaster, but we do have a
new group of folks watching the Servers web.  We would be happy to work
with you to give you access to those entries so you can update them.

-- 
Harlan Stenn 
http://networktimefoundation.org - be a member!

Re: Recent NTP pool traffic increase (update)

[FUJIMURA Sho]

Hello,

I know 133.100.9.2 and 133.100.11.8 are listed.
The Server Contact is old information.
So, I sent e-mail to webmas...@ntp.org a few times.
But, I have't received e-mail from them.
I'd like them to change the information.
Is there the person knowing the contact information to ntp.org?

-- 
Sho FUJIMURA
Information Technology Center, Fukuoka University.
8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan


2016-12-23 9:04 GMT+09:00 Ask Bjørn Hansen :
> Hello,
>
> Those servers aren’t (and have never been) part of the NTP Pool - 
> https://www.ntppool.org/en/
>
> If they were you could remove them from the system and over the next hours, 
> days and months the traffic would go away. We also have features to change 
> the relative amount of clients you get (to just get less queries instead of 
> withdrawing from the pool altogether).
>
> Anyway, it looks like your IPs are listed on support.ntp.org as “public 
> servers”, so removing them from there would be step 1. However there’s no 
> working mechanism for you to tell the clients that they should go away after 
> they’ve hard coded your IP in their configuration. (That’s the point of the 
> NTP Pool system really, to let you offer a public service and have a avenue 
> to stop doing it, too).
>
> support.ntp.org appears to be down, but your IPs are listed on the site 
> according to a Google search:
> https://www.google.com/search?q=133.100.9.2+ntp
>
>
> Ask
>
>> On Dec 21, 2016, at 7:13 PM, FUJIMURA Sho  wrote:
>>
>> Hello.
>>
>> I operate the public NTP Service as 133.100.9.2
>> and 133.100.11.8 at Fukuoka University, Japan.
>> I have a lot of trouble with too much NTP traffic from
>> many routers which 133.100.9.2 as default setting of NTP
>> has been set like Tenda or LB-Link etc.
>> So, although I'd like to contact Firmware developpers of these company
>> and would like them to change the default settins,
>> is there the person knowing the contact information?
>>
>> --
>> Sho FUJIMURA
>> Information Technology Center, Fukuoka University.
>> 8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan
>
>
>

Re: Recent NTP pool traffic increase

[Philip Homburg]

>What I mostly meant is that there should be a regulated, industry-wide 
>effort in order to provide a stable and active pool program. With the 
>current models, a protocol that is widely used by commercial devices is 
>being supported by the time and effort of volunteers around the world. 

My employer has a budget 'for the good of the Internet' So I'd suggest that
people involved in NTP (certainly pool) submit proposals.

Likewise, there are country code DNS registries that are non-profits
and have budgets for research, etc. Given that time is important for
DNSSEC, it maybe worth contacting them.

Re: Recent NTP pool traffic increase

[Laurent Dumont]

Sorry if I wasn't being clear.

What I mostly meant is that there should be a regulated, industry-wide 
effort in order to provide a stable and active pool program. With the 
current models, a protocol that is widely used by commercial devices is 
being supported by the time and effort of volunteers around the world. 
Initiative like the NTF and the Pool program are awesome, but I believe 
that NTP is something that shouldn't relegated to the wayside as the 
internet continues to evolve. Reading a bit more, NTF could be 
"upgraded" into a more official role too.


I'm just a bit puzzled that this entire mixup actually happened with the 
modern internet.


Laurent

On 12/22/2016 08:05 PM, Harlan Stenn wrote:

On 12/22/16 4:11 PM, Ask Bjørn Hansen wrote:

On Dec 20, 2016, at 8:02 PM, Harlan Stenn<st...@nwtime.org>
wrote:


On 12/20/16 7:27 PM, Laurent Dumont wrote: To be honest, the fact
that NTP is still something managed by volunteers and not a
regulated entity (a bit like DNS) is mind boggling.

Time *is* managed by regulated entities - the National Time Labs.

That was pretty clearly not what Laurent was talking about.

Not all that clearly, at least to me.


And Network Time Foundation's NTP Project (the reference
implementation for NTP) could do lots more if we had a useful
budget.

Folks pay money for DNS registrations.  There's no revenue stream
around "time".

Help us get enough support to NTF, and we'll have the staff and
infrastructure to do more for folks.

What does the NTF have to do with the NTP Pool (or the “recent NTP
pool traffic increase”)?

Nothing.  But it has to do with companies putting NTP into their
products.  When they do this with problematic configurations, it causes
trouble.  In this case, it was trouble for the Pool project.

The NTP Pool project certainly isn't the place to solve this issue.


The NTP Pool is run by volunteers, as you very well know. Both the
management and DNS system and the thousands of people who contribute
their NTP service to the system. (And we manage on a pretty scarce
budget).

What's your point?

This sort of misconfiguration will happen and the NTP Pool Project
clearly isn't the place to solve this problem overall.  It *is*
something NTF is in a position to address.

And we're almost exclusively an overstretched volunteer organization,
too, as you very well know.

Do we have different goals around this issue?

Re: Recent NTP pool traffic increase

[Harlan Stenn]

On 12/22/16 5:25 PM, Royce Williams wrote:
> On Thu, Dec 22, 2016 at 4:05 PM, Harlan Stenn  wrote:
> 
>> This sort of misconfiguration will happen and the NTP Pool Project
>> clearly isn't the place to solve this problem overall.  It *is*
>> something NTF is in a position to address.
> 
> Harlan, could you be more specific about how NTF can address this?
> Would it require modification of the NTP protocol, or something else?

Network Time Foundation has two projects to directly help here.

One is a Certification and Compliance program.

The other is a project to analyze/produce BCP-compliant ntp.conf files.

Due to limited resources we're making slow progress on each of these.
That's the backwards way of saying we can make much more useful progress
on these and other issues (like the development of NTPv5 and the General
Timestamp API) if we can get useful support.

With more support we can also make more pool servers available.

-- 
Harlan Stenn 
http://networktimefoundation.org - be a member!

Re: Recent NTP pool traffic increase

[Royce Williams]

On Thu, Dec 22, 2016 at 4:05 PM, Harlan Stenn  wrote:

> This sort of misconfiguration will happen and the NTP Pool Project
> clearly isn't the place to solve this problem overall.  It *is*
> something NTF is in a position to address.

Harlan, could you be more specific about how NTF can address this?
Would it require modification of the NTP protocol, or something else?

Royce

Re: Recent NTP pool traffic increase

[Harlan Stenn]

On 12/22/16 4:11 PM, Ask Bjørn Hansen wrote:
>> On Dec 20, 2016, at 8:02 PM, Harlan Stenn <st...@nwtime.org>
>> wrote:
>> 
>>> On 12/20/16 7:27 PM, Laurent Dumont wrote: To be honest, the fact
>>> that NTP is still something managed by volunteers and not a
>>> regulated entity (a bit like DNS) is mind boggling.
>> 
>> Time *is* managed by regulated entities - the National Time Labs.
> 
> That was pretty clearly not what Laurent was talking about.

Not all that clearly, at least to me.

>> And Network Time Foundation's NTP Project (the reference
>> implementation for NTP) could do lots more if we had a useful
>> budget.
>> 
>> Folks pay money for DNS registrations.  There's no revenue stream
>> around "time".
>> 
>> Help us get enough support to NTF, and we'll have the staff and 
>> infrastructure to do more for folks.
> 
> What does the NTF have to do with the NTP Pool (or the “recent NTP
> pool traffic increase”)?

Nothing.  But it has to do with companies putting NTP into their
products.  When they do this with problematic configurations, it causes
trouble.  In this case, it was trouble for the Pool project.

The NTP Pool project certainly isn't the place to solve this issue.

> The NTP Pool is run by volunteers, as you very well know. Both the
> management and DNS system and the thousands of people who contribute
> their NTP service to the system. (And we manage on a pretty scarce
> budget).

What's your point?

This sort of misconfiguration will happen and the NTP Pool Project
clearly isn't the place to solve this problem overall.  It *is*
something NTF is in a position to address.

And we're almost exclusively an overstretched volunteer organization,
too, as you very well know.

Do we have different goals around this issue?

-- 
Harlan Stenn <st...@nwtime.org>
http://networktimefoundation.org - be a member!

Re: Recent NTP pool traffic increase

[Ask Bjørn Hansen]

> On Dec 20, 2016, at 8:02 PM, Harlan Stenn <st...@nwtime.org> wrote:
> 
>> On 12/20/16 7:27 PM, Laurent Dumont wrote:
>> To be honest, the fact that NTP is still something managed by volunteers
>> and not a regulated entity (a bit like DNS) is mind boggling.
> 
> Time *is* managed by regulated entities - the National Time Labs.

That was pretty clearly not what Laurent was talking about.

> And Network Time Foundation's NTP Project (the reference implementation
> for NTP) could do lots more if we had a useful budget.
> 
> Folks pay money for DNS registrations.  There's no revenue stream around
> "time".
> 
> Help us get enough support to NTF, and we'll have the staff and
> infrastructure to do more for folks.

What does the NTF have to do with the NTP Pool (or the “recent NTP pool traffic 
increase”)?

The NTP Pool is run by volunteers, as you very well know. Both the management 
and DNS system and the thousands of people who contribute their NTP service to 
the system. (And we manage on a pretty scarce budget).


Ask

-- 
http://www.askask.com

Re: Recent NTP pool traffic increase (update)

[Ask Bjørn Hansen]

Hello,

Those servers aren’t (and have never been) part of the NTP Pool - 
https://www.ntppool.org/en/

If they were you could remove them from the system and over the next hours, 
days and months the traffic would go away. We also have features to change the 
relative amount of clients you get (to just get less queries instead of 
withdrawing from the pool altogether).

Anyway, it looks like your IPs are listed on support.ntp.org as “public 
servers”, so removing them from there would be step 1. However there’s no 
working mechanism for you to tell the clients that they should go away after 
they’ve hard coded your IP in their configuration. (That’s the point of the NTP 
Pool system really, to let you offer a public service and have a avenue to stop 
doing it, too).

support.ntp.org appears to be down, but your IPs are listed on the site 
according to a Google search:
https://www.google.com/search?q=133.100.9.2+ntp


Ask

> On Dec 21, 2016, at 7:13 PM, FUJIMURA Sho  wrote:
> 
> Hello.
> 
> I operate the public NTP Service as 133.100.9.2
> and 133.100.11.8 at Fukuoka University, Japan.
> I have a lot of trouble with too much NTP traffic from
> many routers which 133.100.9.2 as default setting of NTP
> has been set like Tenda or LB-Link etc.
> So, although I'd like to contact Firmware developpers of these company
> and would like them to change the default settins,
> is there the person knowing the contact information?
> 
> -- 
> Sho FUJIMURA
> Information Technology Center, Fukuoka University.
> 8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan

Re: Recent NTP pool traffic increase (update)

[FUJIMURA Sho]

Hello.

I operate the public NTP Service as 133.100.9.2
and 133.100.11.8 at Fukuoka University, Japan.
I have a lot of trouble with too much NTP traffic from
many routers which 133.100.9.2 as default setting of NTP
has been set like Tenda or LB-Link etc.
So, although I'd like to contact Firmware developpers of these company
and would like them to change the default settins,
is there the person knowing the contact information?

-- 
Sho FUJIMURA
Information Technology Center, Fukuoka University.
8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan


2016-12-21 18:36 GMT+09:00 Denys Fedoryshchenko :
> Hello,
>
> I'm not sure i should continue to CC nanog, if someone interested to be in
> CC for further updates this story please let me know.
>
> TP-Link not related, it was misunderstanding or wrong customer report.
>
> Tenda routers i believe most of cheap models are affected by this problem.
> On ISPs i have access i see too many of them sending requests to 133.100.9.2
> (if it is unreachable, repeating each 10 seconds), this particular ip seems
> hardcoded there. I am sure as soon as your server is down, way they coded -
> it will make all this routers to ddos this particular ip, repeating NTP
> queries very often without any backoff/protection mechanism.
> Particular model i tested - W308R / Wireless N300 Home Router, but i believe
> many models are affected.
> Firmware: System Version: 5.0.7.53_en hw version : v1.0
>
> Another possible vendor is LB-Link, but i dont have yet any info from
> customers who own them.
>
> On 2016-12-21 11:00, FUJIMURA Sho wrote:
>>
>> Hello.
>>
>> I'm Sho FUJIMURA.
>> Thank you for your information.
>> I operate the public NTP Services as 133.100.9.2 and 133.100.11.8.
>> I'd like to reduce the traffic because I have trouble with too much
>> traffic recently.
>> So, I'm interested in the root of the the problem.
>> If possible, would you please tell me the model numbers of Tenda and
>> TP-Link??
>>
>> --
>> Sho FUJIMURA
>> Information Technology Center, Fukuoka University.
>> 8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan
>>
>> fujim...@fukuoka-u.ac.jp
>>
>> 2016-12-20 5:33 GMT+09:00 Denys Fedoryshchenko :
>>
>>> I'm not sure if this issue relevant to discussed topic, Tenda
>>> routers here for a while on market, and i think i noticed this issue
>>> just now,
>>> because NTP servers they are using supposedly for healthcheck went
>>> down (or NTP owners blocked ISP's i support, due such routers).
>>>
>>> At least after checking numerous users, i believe Tenda hardcoded
>>> those NTP IPs. What worsen issue, that in Lebanon several times per
>>> day, for example at 18pm - short electricity cutoff,
>>> and majority of users routers will reboot and surely reconnect, so
>>> it will look like a countrywide spike in NTP traffic.
>>>
>>> I checked for a 10min also this NTP ips in dns responses, none of
>>> thousands of users tried to resolve any name with them over any DNS
>>> server, so i conclude they are hardcoded somewhere in firmware.
>>>
>>> Here is traffic of Tenda router after reconnecting (but not full
>>> powercycle, i dont have it in my hands). But as you can see, no DNS
>>> resolution attempts:
>>>
>>> 20:15:59.305739 PPPoE  [ses 0x1483] CHAP, Success (0x03), id 1, Msg
>>> S=XX M=Authentication succeeded
>>> 20:15:59.306100 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1,
>>> length 12
>>> 20:15:59.317840 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1,
>>> length 24
>>> 20:15:59.317841 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 1,
>>> length 12
>>> 20:15:59.317867 PPPoE  [ses 0x1483] IPCP, Conf-Nack (0x03), id 1,
>>> length 18
>>> 20:15:59.325253 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 2,
>>> length 24
>>> 20:15:59.325273 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 2,
>>> length 24
>>> 20:15:59.335589 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
>>> 133.100.9.2.123: NTPv3, Client, length 48
>>> 20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
>>> 192.5.41.41.123: NTPv3, Client, length 48
>>> 20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
>>> 192.5.41.40.123: NTPv3, Client, length 48
>>>
>>> Here is example of Tenda traffic if it is unable to reach
>>> destination, it repeats request each 10 seconds endlessly, my guess
>>> they are using ntp to show
>>> status of internet connection.
>>> So, now that NTP servers getting quite significant DDoS such way.
>>>
>>> 19:57:52.162863 IP (tos 0x0, ttl 64, id 38515, offset 0, flags
>>> [none], proto UDP (17), length 76)
>>> 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length
>>> 48
>>> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
>>> 0 (1s), precision 0
>>> Root Delay: 0.00, Root dispersion: 0.00,
>>> Reference-ID: (unspec)
>>> Reference Timestamp:  0.0
>>> Originator Timestamp: 0.0
>>> Receive Timestamp:0.0
>>> Transmit Timestamp:   3691177063.0 (2016/12/19
>>> 22:57:43)
>>> 

Re: Recent NTP pool traffic increase (update)

[FUJIMURA Sho]

Hello.

I'm Sho FUJIMURA.
I operate the public NTP Services as 133.100.9.2 and 133.100.11.8.
I'd like to reduce the traffic because I have trouble with too much
traffic recently.
So, I'm interested in the root of the the problem.
If possible, would you please tell me the model numbers of Tenda and TP-Link??

-- 
Sho FUJIMURA
Information Technology Center, Fukuoka University.
8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan


2016-12-20 5:33 GMT+09:00 Denys Fedoryshchenko :
> I'm not sure if this issue relevant to discussed topic, Tenda routers here
> for a while on market, and i think i noticed this issue just now,
> because NTP servers they are using supposedly for healthcheck went down (or
> NTP owners blocked ISP's i support, due such routers).
>
> At least after checking numerous users, i believe Tenda hardcoded those NTP
> IPs. What worsen issue, that in Lebanon several times per day, for example
> at 18pm - short electricity cutoff,
> and majority of users routers will reboot and surely reconnect, so it will
> look like a countrywide spike in NTP traffic.
>
> I checked for a 10min also this NTP ips in dns responses, none of thousands
> of users tried to resolve any name with them over any DNS server, so i
> conclude they are hardcoded somewhere in firmware.
>
> Here is traffic of Tenda router after reconnecting (but not full powercycle,
> i dont have it in my hands). But as you can see, no DNS resolution attempts:
>
> 20:15:59.305739 PPPoE  [ses 0x1483] CHAP, Success (0x03), id 1, Msg S=XX
> M=Authentication succeeded
> 20:15:59.306100 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1, length
> 12
> 20:15:59.317840 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1, length
> 24
> 20:15:59.317841 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 1, length 12
> 20:15:59.317867 PPPoE  [ses 0x1483] IPCP, Conf-Nack (0x03), id 1, length 18
> 20:15:59.325253 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 2, length
> 24
> 20:15:59.325273 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 2, length 24
> 20:15:59.335589 PPPoE  [ses 0x1483] IP 172.17.49.245.123 > 133.100.9.2.123:
> NTPv3, Client, length 48
> 20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 > 192.5.41.41.123:
> NTPv3, Client, length 48
> 20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 > 192.5.41.40.123:
> NTPv3, Client, length 48
>
>
> Here is example of Tenda traffic if it is unable to reach destination, it
> repeats request each 10 seconds endlessly, my guess they are using ntp to
> show
> status of internet connection.
> So, now that NTP servers getting quite significant DDoS such way.
>
> 19:57:52.162863 IP (tos 0x0, ttl 64, id 38515, offset 0, flags [none], proto
> UDP (17), length 76)
> 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length 48
> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s),
> precision 0
> Root Delay: 0.00, Root dispersion: 0.00, Reference-ID:
> (unspec)
>   Reference Timestamp:  0.0
>   Originator Timestamp: 0.0
>   Receive Timestamp:0.0
>   Transmit Timestamp:   3691177063.0 (2016/12/19 22:57:43)
> Originator - Receive Timestamp:  0.0
> Originator - Transmit Timestamp: 3691177063.0
> (2016/12/19 22:57:43)
> 19:57:52.163277 IP (tos 0x0, ttl 64, id 38516, offset 0, flags [none], proto
> UDP (17), length 76)
> 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length 48
> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s),
> precision 0
> Root Delay: 0.00, Root dispersion: 0.00, Reference-ID:
> (unspec)
>   Reference Timestamp:  0.0
>   Originator Timestamp: 0.0
>   Receive Timestamp:0.0
>   Transmit Timestamp:   3691177063.0 (2016/12/19 22:57:43)
> Originator - Receive Timestamp:  0.0
> Originator - Transmit Timestamp: 3691177063.0
> (2016/12/19 22:57:43)
> 19:57:52.164435 IP (tos 0x0, ttl 64, id 38517, offset 0, flags [none], proto
> UDP (17), length 76)
> 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length 48
> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s),
> precision 0
> Root Delay: 0.00, Root dispersion: 0.00, Reference-ID:
> (unspec)
>   Reference Timestamp:  0.0
>   Originator Timestamp: 0.0
>   Receive Timestamp:0.0
>   Transmit Timestamp:   3691177063.0 (2016/12/19 22:57:43)
> Originator - Receive Timestamp:  0.0
> Originator - Transmit Timestamp: 3691177063.0
> (2016/12/19 22:57:43)
> 19:58:02.164781 IP (tos 0x0, ttl 64, id 38518, offset 0, flags [none], proto
> UDP (17), length 76)
> 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length 48
> Client, Leap indicator:  

Re: Recent NTP pool traffic increase (update)

[Denys Fedoryshchenko]

Hello,

I'm not sure i should continue to CC nanog, if someone interested to be 
in CC for further updates this story please let me know.


TP-Link not related, it was misunderstanding or wrong customer report.

Tenda routers i believe most of cheap models are affected by this 
problem.
On ISPs i have access i see too many of them sending requests to 
133.100.9.2 (if it is unreachable, repeating each 10 seconds), this 
particular ip seems hardcoded there. I am sure as soon as your server is 
down, way they coded - it will make all this routers to ddos this 
particular ip, repeating NTP queries very often without any 
backoff/protection mechanism.
Particular model i tested - W308R / Wireless N300 Home Router, but i 
believe many models are affected.

Firmware: System Version: 5.0.7.53_en hw version : v1.0

Another possible vendor is LB-Link, but i dont have yet any info from 
customers who own them.


On 2016-12-21 11:00, FUJIMURA Sho wrote:

Hello.

I'm Sho FUJIMURA.
Thank you for your information.
I operate the public NTP Services as 133.100.9.2 and 133.100.11.8.
I'd like to reduce the traffic because I have trouble with too much
traffic recently.
So, I'm interested in the root of the the problem.
If possible, would you please tell me the model numbers of Tenda and
TP-Link??

--
Sho FUJIMURA
Information Technology Center, Fukuoka University.
8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan

fujim...@fukuoka-u.ac.jp

2016-12-20 5:33 GMT+09:00 Denys Fedoryshchenko :


I'm not sure if this issue relevant to discussed topic, Tenda
routers here for a while on market, and i think i noticed this issue
just now,
because NTP servers they are using supposedly for healthcheck went
down (or NTP owners blocked ISP's i support, due such routers).

At least after checking numerous users, i believe Tenda hardcoded
those NTP IPs. What worsen issue, that in Lebanon several times per
day, for example at 18pm - short electricity cutoff,
and majority of users routers will reboot and surely reconnect, so
it will look like a countrywide spike in NTP traffic.

I checked for a 10min also this NTP ips in dns responses, none of
thousands of users tried to resolve any name with them over any DNS
server, so i conclude they are hardcoded somewhere in firmware.

Here is traffic of Tenda router after reconnecting (but not full
powercycle, i dont have it in my hands). But as you can see, no DNS
resolution attempts:

20:15:59.305739 PPPoE  [ses 0x1483] CHAP, Success (0x03), id 1, Msg
S=XX M=Authentication succeeded
20:15:59.306100 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1,
length 12
20:15:59.317840 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1,
length 24
20:15:59.317841 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 1,
length 12
20:15:59.317867 PPPoE  [ses 0x1483] IPCP, Conf-Nack (0x03), id 1,
length 18
20:15:59.325253 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 2,
length 24
20:15:59.325273 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 2,
length 24
20:15:59.335589 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
133.100.9.2.123: NTPv3, Client, length 48
20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
192.5.41.41.123: NTPv3, Client, length 48
20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
192.5.41.40.123: NTPv3, Client, length 48

Here is example of Tenda traffic if it is unable to reach
destination, it repeats request each 10 seconds endlessly, my guess
they are using ntp to show
status of internet connection.
So, now that NTP servers getting quite significant DDoS such way.

19:57:52.162863 IP (tos 0x0, ttl 64, id 38515, offset 0, flags
[none], proto UDP (17), length 76)
172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length
48
Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
0 (1s), precision 0
Root Delay: 0.00, Root dispersion: 0.00,
Reference-ID: (unspec)
Reference Timestamp:  0.0
Originator Timestamp: 0.0
Receive Timestamp:0.0
Transmit Timestamp:   3691177063.0 (2016/12/19
22:57:43)
Originator - Receive Timestamp:  0.0
Originator - Transmit Timestamp: 3691177063.0
(2016/12/19 22:57:43)
19:57:52.163277 IP (tos 0x0, ttl 64, id 38516, offset 0, flags
[none], proto UDP (17), length 76)
172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length
48
Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
0 (1s), precision 0
Root Delay: 0.00, Root dispersion: 0.00,
Reference-ID: (unspec)
Reference Timestamp:  0.0
Originator Timestamp: 0.0
Receive Timestamp:0.0
Transmit Timestamp:   3691177063.0 (2016/12/19
22:57:43)
Originator - Receive Timestamp:  0.0
Originator - Transmit Timestamp: 3691177063.0
(2016/12/19 22:57:43)
19:57:52.164435 IP (tos 0x0, ttl 64, id 38517, offset 0, flags
[none], proto UDP (17), length 76)
172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length
48
Client, Leap indicator:  (0), Stratum 0 

Re: Recent NTP pool traffic increase

[Damian Menscher via NANOG]

On Tue, Dec 20, 2016 at 6:41 PM, Keenan Tims  wrote:

> In a similar vein, I've always been curious what the ratio Google sees of
> ICMP echo vs. DNS traffic to 8.8.8.8 is...
>

The more fun question is how many pagers would go off around the world if
Google stopped responding to ICMP echo.

Damian

Re: Recent NTP pool traffic increase

[Harlan Stenn]

On 12/20/16 9:21 PM, Royce Williams wrote:
> On Tue, Dec 20, 2016 at 8:19 PM, Royce Williams  
> wrote:
>> On Tue, Dec 20, 2016 at 8:04 PM, Yury Shefer  wrote:
>>>
>>> Google announced public NTP service some time ago:
>>> https://developers.google.com/time/
>>
>> Leap smearing does look interesting as way to sidestep the
>> potentially-jarring leap-second problem ... but a note of caution.
>>
>> I've had multiple time geeks tell me that leap-smearing is pretty
>> different from strict-RFC NTP, and Google themselves say on that page:
>>
>> "We recommend that you don’t configure Google Public NTP together with
>> non-leap-smearing NTP servers."
>>
>> So it looks like we shouldn't mix and match. And since most folks
>> should probably want some heterogeneity in their NTP, it may be a
>> little premature to jump on the leap-smear bandwagon just yet.
>>
>> I'm vague on the details, so I could be wrong.
> 
> This is informative:
> 
> https://docs.ntpsec.org/latest/leapsmear.html
> 
>> Does anyone know of any other (non Google) leap-smearing NTP implementations?

The NTP Project has had a leap-smear implementation for a while.

We also have a proposal for a REFID that indicates the provided time is
a leap-smear time, and Network Time Foundation is working on a new
timestamp format and API that will easily allow time exchange between
systems using different timescales.

-- 
Harlan Stenn 
http://networktimefoundation.org - be a member!

Re: Recent NTP pool traffic increase

[Royce Williams]

On Tue, Dec 20, 2016 at 8:19 PM, Royce Williams  wrote:
> On Tue, Dec 20, 2016 at 8:04 PM, Yury Shefer  wrote:
>>
>> Google announced public NTP service some time ago:
>> https://developers.google.com/time/
>
> Leap smearing does look interesting as way to sidestep the
> potentially-jarring leap-second problem ... but a note of caution.
>
> I've had multiple time geeks tell me that leap-smearing is pretty
> different from strict-RFC NTP, and Google themselves say on that page:
>
> "We recommend that you don’t configure Google Public NTP together with
> non-leap-smearing NTP servers."
>
> So it looks like we shouldn't mix and match. And since most folks
> should probably want some heterogeneity in their NTP, it may be a
> little premature to jump on the leap-smear bandwagon just yet.
>
> I'm vague on the details, so I could be wrong.

This is informative:

https://docs.ntpsec.org/latest/leapsmear.html

> Does anyone know of any other (non Google) leap-smearing NTP implementations?

Royce

Re: Recent NTP pool traffic increase

[Royce Williams]

On Tue, Dec 20, 2016 at 8:04 PM, Yury Shefer  wrote:
>
> Google announced public NTP service some time ago:
> https://developers.google.com/time/

Leap smearing does look interesting as way to sidestep the
potentially-jarring leap-second problem ... but a note of caution.

I've had multiple time geeks tell me that leap-smearing is pretty
different from strict-RFC NTP, and Google themselves say on that page:

"We recommend that you don’t configure Google Public NTP together with
non-leap-smearing NTP servers."

So it looks like we shouldn't mix and match. And since most folks
should probably want some heterogeneity in their NTP, it may be a
little premature to jump on the leap-smear bandwagon just yet.

I'm vague on the details, so I could be wrong.

Does anyone know of any other (non Google) leap-smearing NTP implementations?

Royce

Re: Recent NTP pool traffic increase

[Yury Shefer]

Google announced public NTP service some time ago:
https://developers.google.com/time/


On Tue, Dec 20, 2016 at 7:29 PM Laurent Dumont <ad...@coldnorthadmin.com>
wrote:

> I do think that the point of the Pool network is to be used by both
>
> consumers and vendors. And as mentioned before, there is a process if
>
> you are a vendor and want to use the pool within a commercial product. I
>
> have 3 NTP servers running and I don't really care who is using it.
>
>
>
> That said, setting up your own infrastructure is also worth considering
>
> if it's a business critical feature. I assume that a Snapchat app that
>
> fails to have accurate time or correct itself could be abused.
>
>
>
> To be honest, the fact that NTP is still something managed by volunteers
>
> and not a regulated entity (a bit like DNS) is mind boggling.
>
>
>
> On 12/20/2016 09:41 PM, Keenan Tims wrote:
>
> > Better for whom? I'm sure all mobile operating systems provide some
>
> > access to time, with a least 'seconds' resolution. If an app deems
>
> > this time source untrustworthy for some reason, I don't think the
>
> > reasonable response is to make independent time requests from a
>
> > volunteer-operated pool for public servers designed for host
>
> > synchronization. Particularly on mobile, the compartmentalization of
>
> > applications means that this 'better' time will only be accessible to
>
> > one application, and many applications may have this 'better time'
>
> > requirement. These developers should be lobbying Apple and Google for
>
> > better time, if they need it, not making many millions of calls to the
>
> > NTP pool. To make things worse, I'm fairly sure that Apple's 'no
>
> > background tasks' policy means that an application can't *maintain*
>
> > its sense of time, so it would not surprise me if it fires off NTP
>
> > requests every time it is focused, further compounding the burden.
>
> >
>
> > Time is already available, and having every application query for its
>
> > own time against a public resource doesn't seem very friendly. It
>
> > certainly doesn't scale. If they are unsuccessful lobbying the OS, why
>
> > not trust the time provided by the API calls they are surely doing to
>
> > their own servers? Most HTTP responses include a timestamp; surely
>
> > this is good enough for expiring Snaps. Or at least operate their own
>
> > NTP infrastructure.
>
> >
>
> > I'm sure that Snap had no malicious intent and commend them for their
>
> > quick and appropriate response once the issue was identified, but I
>
> > don't think this behaviour is very defensible. I for one was not
>
> > harmed by the ~10x increase in load and traffic on my NTP pool node,
>
> > but the 100x increase if a handful of similar apps decided they 'need'
>
> > more accurate time than the OS provides would be cause for concern,
>
> > and I suspect a great many pool nodes would simply disappear,
>
> > compounding the problem. Please make use of these and similar services
>
> > as they are designed to be used, and as efficiently as possible,
>
> > especially if you are responsible for millions of users / machines.
>
> >
>
> > In a similar vein, I've always been curious what the ratio Google sees
>
> > of ICMP echo vs. DNS traffic to 8.8.8.8 is...
>
> >
>
> > Keenan
>
> >
>
> >
>
> > On 2016-12-20 18:16, Tim Raphael wrote:
>
> >> Exactly,
>
> >>
>
> >> Also they’re across Android and iOS and getting parity of operations
>
> >> across those two OSs isn’t easy.
>
> >> Better to just embed what they need inside the app if it is
>
> >> specialised enough.
>
> >>
>
> >> - Tim
>
> >>
>
> >>> On 21 Dec. 2016, at 10:13 am, Emille Blanc
>
> >>> <emi...@abccommunications.com> wrote:
>
> >>>
>
> >>> Perhaps the host OS' to which snapchat caters, don't all have a
>
> >>> devent ntp subststem available?
>
> >>> I have vague recollections of some other software (I'm sure we all
>
> >>> know which) implemented it's own malloc layer for every system it
>
> >>> ran on, for less trivial reasons. ;)
>
> >>>
>
> >>> 
>
> >>> From: NANOG [nanog-boun...@nanog.org] On Behalf Of Tim Raphael
>
> >>> [raphael.timo...@gmail.com]
>
> >>> Sent: Tuesday, December 20, 2016 5

Re: Recent NTP pool traffic increase

[Harlan Stenn]

On 12/20/16 7:27 PM, Laurent Dumont wrote:
> I do think that the point of the Pool network is to be used by both
> consumers and vendors. And as mentioned before, there is a process if
> you are a vendor and want to use the pool within a commercial product. I
> have 3 NTP servers running and I don't really care who is using it.
> 
> That said, setting up your own infrastructure is also worth considering
> if it's a business critical feature. I assume that a Snapchat app that
> fails to have accurate time or correct itself could be abused.
> 
> To be honest, the fact that NTP is still something managed by volunteers
> and not a regulated entity (a bit like DNS) is mind boggling.

Time *is* managed by regulated entities - the National Time Labs.

And Network Time Foundation's NTP Project (the reference implementation
for NTP) could do lots more if we had a useful budget.

Folks pay money for DNS registrations.  There's no revenue stream around
"time".

Help us get enough support to NTF, and we'll have the staff and
infrastructure to do more for folks.
-- 
Harlan Stenn 
http://networktimefoundation.org - be a member!

Re: Recent NTP pool traffic increase

[Valdis . Kletnieks]

On Tue, 20 Dec 2016 18:41:37 -0800, Keenan Tims said:
> Better for whom? I'm sure all mobile operating systems provide some
> access to time, with a least 'seconds' resolution. If an app deems this
> time source untrustworthy for some reason, I don't think the reasonable
> response is to make independent time requests from a volunteer-operated
> pool for public servers designed for host synchronization.

This is possibly at least partly due to "dependency hell".
For a worked example from earlier this year:

http://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/

So a lot of people had their stuff blow up, even though their code
called left_pad exactly noplace



pgpQ6G_Js6GBs.pgp
Description: PGP signature

Re: Recent NTP pool traffic increase

[Laurent Dumont]

I do think that the point of the Pool network is to be used by both 
consumers and vendors. And as mentioned before, there is a process if 
you are a vendor and want to use the pool within a commercial product. I 
have 3 NTP servers running and I don't really care who is using it.


That said, setting up your own infrastructure is also worth considering 
if it's a business critical feature. I assume that a Snapchat app that 
fails to have accurate time or correct itself could be abused.


To be honest, the fact that NTP is still something managed by volunteers 
and not a regulated entity (a bit like DNS) is mind boggling.


On 12/20/2016 09:41 PM, Keenan Tims wrote:
Better for whom? I'm sure all mobile operating systems provide some 
access to time, with a least 'seconds' resolution. If an app deems 
this time source untrustworthy for some reason, I don't think the 
reasonable response is to make independent time requests from a 
volunteer-operated pool for public servers designed for host 
synchronization. Particularly on mobile, the compartmentalization of 
applications means that this 'better' time will only be accessible to 
one application, and many applications may have this 'better time' 
requirement. These developers should be lobbying Apple and Google for 
better time, if they need it, not making many millions of calls to the 
NTP pool. To make things worse, I'm fairly sure that Apple's 'no 
background tasks' policy means that an application can't *maintain* 
its sense of time, so it would not surprise me if it fires off NTP 
requests every time it is focused, further compounding the burden.


Time is already available, and having every application query for its 
own time against a public resource doesn't seem very friendly. It 
certainly doesn't scale. If they are unsuccessful lobbying the OS, why 
not trust the time provided by the API calls they are surely doing to 
their own servers? Most HTTP responses include a timestamp; surely 
this is good enough for expiring Snaps. Or at least operate their own 
NTP infrastructure.


I'm sure that Snap had no malicious intent and commend them for their 
quick and appropriate response once the issue was identified, but I 
don't think this behaviour is very defensible. I for one was not 
harmed by the ~10x increase in load and traffic on my NTP pool node, 
but the 100x increase if a handful of similar apps decided they 'need' 
more accurate time than the OS provides would be cause for concern, 
and I suspect a great many pool nodes would simply disappear, 
compounding the problem. Please make use of these and similar services 
as they are designed to be used, and as efficiently as possible, 
especially if you are responsible for millions of users / machines.


In a similar vein, I've always been curious what the ratio Google sees 
of ICMP echo vs. DNS traffic to 8.8.8.8 is...


Keenan


On 2016-12-20 18:16, Tim Raphael wrote:

Exactly,

Also they’re across Android and iOS and getting parity of operations 
across those two OSs isn’t easy.
Better to just embed what they need inside the app if it is 
specialised enough.


- Tim

On 21 Dec. 2016, at 10:13 am, Emille Blanc 
<emi...@abccommunications.com> wrote:


Perhaps the host OS' to which snapchat caters, don't all have a 
devent ntp subststem available?
I have vague recollections of some other software (I'm sure we all 
know which) implemented it's own malloc layer for every system it 
ran on, for less trivial reasons. ;)



From: NANOG [nanog-boun...@nanog.org] On Behalf Of Tim Raphael 
[raphael.timo...@gmail.com]

Sent: Tuesday, December 20, 2016 5:34 PM
To: Gary E. Miller
Cc: nanog@nanog.org
Subject: Re: Recent NTP pool traffic increase

This was my thought actually, Apple does offer some time services as 
part of the OS but it’s becoming common with larger / more popular 
apps to provide some of these services internally.
Look at the FB app for example, there are a lot of “system” things 
they do themselves due to the ability to control specifics. Users 
don’t want to have to install a second “specialised app” for this 
either.


With regard to an ephemeral chat app requiring time sync, I can 
think of quite a few use cases and mechanisms in the app that might 
require time services.


- Tim



On 21 Dec. 2016, at 9:26 am, Gary E. Miller <g...@rellim.com> wrote:

Yo valdis.kletni...@vt.edu!

On Tue, 20 Dec 2016 20:20:48 -0500
valdis.kletni...@vt.edu wrote:


On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:

Mostly out of curiosity, what was the reason for the change in the
Snapchat code, and what plans does Snap have for whatever reason
the NTP change was put in place?

 From other comments in the thread, it sounds like the app was simply
linked against a broken version of a library

But why is a chat app doing NTP at all?  it should rely on the OS, or
a specialized app, to keep local time accurate

Re: Recent NTP pool traffic increase

[Keenan Tims]

Better for whom? I'm sure all mobile operating systems provide some 
access to time, with a least 'seconds' resolution. If an app deems this 
time source untrustworthy for some reason, I don't think the reasonable 
response is to make independent time requests from a volunteer-operated 
pool for public servers designed for host synchronization. Particularly 
on mobile, the compartmentalization of applications means that this 
'better' time will only be accessible to one application, and many 
applications may have this 'better time' requirement. These developers 
should be lobbying Apple and Google for better time, if they need it, 
not making many millions of calls to the NTP pool. To make things worse, 
I'm fairly sure that Apple's 'no background tasks' policy means that an 
application can't *maintain* its sense of time, so it would not surprise 
me if it fires off NTP requests every time it is focused, further 
compounding the burden.


Time is already available, and having every application query for its 
own time against a public resource doesn't seem very friendly. It 
certainly doesn't scale. If they are unsuccessful lobbying the OS, why 
not trust the time provided by the API calls they are surely doing to 
their own servers? Most HTTP responses include a timestamp; surely this 
is good enough for expiring Snaps. Or at least operate their own NTP 
infrastructure.


I'm sure that Snap had no malicious intent and commend them for their 
quick and appropriate response once the issue was identified, but I 
don't think this behaviour is very defensible. I for one was not harmed 
by the ~10x increase in load and traffic on my NTP pool node, but the 
100x increase if a handful of similar apps decided they 'need' more 
accurate time than the OS provides would be cause for concern, and I 
suspect a great many pool nodes would simply disappear, compounding the 
problem. Please make use of these and similar services as they are 
designed to be used, and as efficiently as possible, especially if you 
are responsible for millions of users / machines.


In a similar vein, I've always been curious what the ratio Google sees 
of ICMP echo vs. DNS traffic to 8.8.8.8 is...


Keenan


On 2016-12-20 18:16, Tim Raphael wrote:

Exactly,

Also they’re across Android and iOS and getting parity of operations across 
those two OSs isn’t easy.
Better to just embed what they need inside the app if it is specialised enough.

- Tim


On 21 Dec. 2016, at 10:13 am, Emille Blanc <emi...@abccommunications.com> wrote:

Perhaps the host OS' to which snapchat caters, don't all have a devent ntp 
subststem available?
I have vague recollections of some other software (I'm sure we all know which) 
implemented it's own malloc layer for every system it ran on, for less trivial 
reasons. ;)


From: NANOG [nanog-boun...@nanog.org] On Behalf Of Tim Raphael 
[raphael.timo...@gmail.com]
Sent: Tuesday, December 20, 2016 5:34 PM
To: Gary E. Miller
Cc: nanog@nanog.org
Subject: Re: Recent NTP pool traffic increase

This was my thought actually, Apple does offer some time services as part of 
the OS but it’s becoming common with larger / more popular apps to provide some 
of these services internally.
Look at the FB app for example, there are a lot of “system” things they do 
themselves due to the ability to control specifics. Users don’t want to have to 
install a second “specialised app” for this either.

With regard to an ephemeral chat app requiring time sync, I can think of quite 
a few use cases and mechanisms in the app that might require time services.

- Tim



On 21 Dec. 2016, at 9:26 am, Gary E. Miller <g...@rellim.com> wrote:

Yo valdis.kletni...@vt.edu!

On Tue, 20 Dec 2016 20:20:48 -0500
valdis.kletni...@vt.edu wrote:


On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:

Mostly out of curiosity, what was the reason for the change in the
Snapchat code, and what plans does Snap have for whatever reason
the NTP change was put in place?

 From other comments in the thread, it sounds like the app was simply
linked against a broken version of a library

But why is a chat app doing NTP at all?  it should rely on the OS, or
a specialized app, to keep local time accurate.

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
  g...@rellim.com  Tel:+1 541 382 8588

Re: Recent NTP pool traffic increase

[Jad Boutros via NANOG]

The Snapchat mobile app does not currently have a need to talk with NTP
servers, it was an error. Also, this afternoon we spun off NTP server
instances in Australia and South America to help with the load.

Thanks,
Jad

On Tue, Dec 20, 2016 at 5:34 PM, Tim Raphael 
wrote:

> This was my thought actually, Apple does offer some time services as part
> of the OS but it’s becoming common with larger / more popular apps to
> provide some of these services internally.
> Look at the FB app for example, there are a lot of “system” things they do
> themselves due to the ability to control specifics. Users don’t want to
> have to install a second “specialised app” for this either.
>
> With regard to an ephemeral chat app requiring time sync, I can think of
> quite a few use cases and mechanisms in the app that might require time
> services.
>
> - Tim
>
>
> > On 21 Dec. 2016, at 9:26 am, Gary E. Miller  wrote:
> >
> > Yo valdis.kletni...@vt.edu!
> >
> > On Tue, 20 Dec 2016 20:20:48 -0500
> > valdis.kletni...@vt.edu wrote:
> >
> >> On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:
> >>> Mostly out of curiosity, what was the reason for the change in the
> >>> Snapchat code, and what plans does Snap have for whatever reason
> >>> the NTP change was put in place?
> >>
> >> From other comments in the thread, it sounds like the app was simply
> >> linked against a broken version of a library
> >
> > But why is a chat app doing NTP at all?  it should rely on the OS, or
> > a specialized app, to keep local time accurate.
> >
> > RGDS
> > GARY
> > 
> ---
> > Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
> >   g...@rellim.com  Tel:+1 541 382 8588
>
>

Re: Recent NTP pool traffic increase

[Tim Raphael]

Exactly,

Also they’re across Android and iOS and getting parity of operations across 
those two OSs isn’t easy. 
Better to just embed what they need inside the app if it is specialised enough.

- Tim

> On 21 Dec. 2016, at 10:13 am, Emille Blanc <emi...@abccommunications.com> 
> wrote:
> 
> Perhaps the host OS' to which snapchat caters, don't all have a devent ntp 
> subststem available?
> I have vague recollections of some other software (I'm sure we all know 
> which) implemented it's own malloc layer for every system it ran on, for less 
> trivial reasons. ;)
> 
> 
> From: NANOG [nanog-boun...@nanog.org] On Behalf Of Tim Raphael 
> [raphael.timo...@gmail.com]
> Sent: Tuesday, December 20, 2016 5:34 PM
> To: Gary E. Miller
> Cc: nanog@nanog.org
> Subject: Re: Recent NTP pool traffic increase
> 
> This was my thought actually, Apple does offer some time services as part of 
> the OS but it’s becoming common with larger / more popular apps to provide 
> some of these services internally.
> Look at the FB app for example, there are a lot of “system” things they do 
> themselves due to the ability to control specifics. Users don’t want to have 
> to install a second “specialised app” for this either.
> 
> With regard to an ephemeral chat app requiring time sync, I can think of 
> quite a few use cases and mechanisms in the app that might require time 
> services.
> 
> - Tim
> 
> 
>> On 21 Dec. 2016, at 9:26 am, Gary E. Miller <g...@rellim.com> wrote:
>> 
>> Yo valdis.kletni...@vt.edu!
>> 
>> On Tue, 20 Dec 2016 20:20:48 -0500
>> valdis.kletni...@vt.edu wrote:
>> 
>>> On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:
>>>> Mostly out of curiosity, what was the reason for the change in the
>>>> Snapchat code, and what plans does Snap have for whatever reason
>>>> the NTP change was put in place?
>>> 
>>> From other comments in the thread, it sounds like the app was simply
>>> linked against a broken version of a library
>> 
>> But why is a chat app doing NTP at all?  it should rely on the OS, or
>> a specialized app, to keep local time accurate.
>> 
>> RGDS
>> GARY
>> ---
>> Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
>>  g...@rellim.com  Tel:+1 541 382 8588

RE: Recent NTP pool traffic increase

[Emille Blanc]

Perhaps the host OS' to which snapchat caters, don't all have a devent ntp 
subststem available?
I have vague recollections of some other software (I'm sure we all know which) 
implemented it's own malloc layer for every system it ran on, for less trivial 
reasons. ;)


From: NANOG [nanog-boun...@nanog.org] On Behalf Of Tim Raphael 
[raphael.timo...@gmail.com]
Sent: Tuesday, December 20, 2016 5:34 PM
To: Gary E. Miller
Cc: nanog@nanog.org
Subject: Re: Recent NTP pool traffic increase

This was my thought actually, Apple does offer some time services as part of 
the OS but it’s becoming common with larger / more popular apps to provide some 
of these services internally.
Look at the FB app for example, there are a lot of “system” things they do 
themselves due to the ability to control specifics. Users don’t want to have to 
install a second “specialised app” for this either.

With regard to an ephemeral chat app requiring time sync, I can think of quite 
a few use cases and mechanisms in the app that might require time services.

- Tim


> On 21 Dec. 2016, at 9:26 am, Gary E. Miller <g...@rellim.com> wrote:
>
> Yo valdis.kletni...@vt.edu!
>
> On Tue, 20 Dec 2016 20:20:48 -0500
> valdis.kletni...@vt.edu wrote:
>
>> On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:
>>> Mostly out of curiosity, what was the reason for the change in the
>>> Snapchat code, and what plans does Snap have for whatever reason
>>> the NTP change was put in place?
>>
>> From other comments in the thread, it sounds like the app was simply
>> linked against a broken version of a library
>
> But why is a chat app doing NTP at all?  it should rely on the OS, or
> a specialized app, to keep local time accurate.
>
> RGDS
> GARY
> ---
> Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
>   g...@rellim.com  Tel:+1 541 382 8588

Re: Recent NTP pool traffic increase

[Tim Raphael]

This was my thought actually, Apple does offer some time services as part of 
the OS but it’s becoming common with larger / more popular apps to provide some 
of these services internally.
Look at the FB app for example, there are a lot of “system” things they do 
themselves due to the ability to control specifics. Users don’t want to have to 
install a second “specialised app” for this either.

With regard to an ephemeral chat app requiring time sync, I can think of quite 
a few use cases and mechanisms in the app that might require time services.

- Tim


> On 21 Dec. 2016, at 9:26 am, Gary E. Miller  wrote:
> 
> Yo valdis.kletni...@vt.edu!
> 
> On Tue, 20 Dec 2016 20:20:48 -0500
> valdis.kletni...@vt.edu wrote:
> 
>> On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:
>>> Mostly out of curiosity, what was the reason for the change in the
>>> Snapchat code, and what plans does Snap have for whatever reason
>>> the NTP change was put in place?  
>> 
>> From other comments in the thread, it sounds like the app was simply
>> linked against a broken version of a library
> 
> But why is a chat app doing NTP at all?  it should rely on the OS, or
> a specialized app, to keep local time accurate.
> 
> RGDS
> GARY
> ---
> Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
>   g...@rellim.com  Tel:+1 541 382 8588

Re: Recent NTP pool traffic increase

[Gary E. Miller]

Yo valdis.kletni...@vt.edu!

On Tue, 20 Dec 2016 20:20:48 -0500
valdis.kletni...@vt.edu wrote:

> On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:
> > Mostly out of curiosity, what was the reason for the change in the
> > Snapchat code, and what plans does Snap have for whatever reason
> > the NTP change was put in place?  
> 
> From other comments in the thread, it sounds like the app was simply
> linked against a broken version of a library

But why is a chat app doing NTP at all?  it should rely on the OS, or
a specialized app, to keep local time accurate.

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588


pgpzwM706_yaZ.pgp
Description: OpenPGP digital signature

Re: Recent NTP pool traffic increase

[Valdis . Kletnieks]

On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:
> Mostly out of curiosity, what was the reason for the change in the Snapchat
> code, and what plans does Snap have for whatever reason the NTP change was
> put in place?

>From other comments in the thread, it sounds like the app was simply linked
against a broken version of a library


pgpseE9KQnGCG.pgp
Description: PGP signature

Re: Recent NTP pool traffic increase

[Peter Beckman]

Mostly out of curiosity, what was the reason for the change in the Snapchat
code, and what plans does Snap have for whatever reason the NTP change was
put in place?

Beckman

On Tue, 20 Dec 2016, Jad Boutros via NANOG wrote:


Immediately after being notified that our latest iOS release was causing
problems with NTP traffic, we started working to disable the offending code
in v9.45. We submitted a new mobile release to the Apple App Store earlier
this morning for their review, which should disable these NTP requests. We
are hoping Apple will be able to review this release in time before the
holiday break, and we have stressed its urgency. When the release does get
approved, we should very quickly begin to see a decrease in NTP traffic
from our app as users start upgrading to the new release.

We deeply regret this situation, and we will post an update here once we
hear back from Apple. We are also open to any suggestions on how we can
help with the present traffic.

On Mon, Dec 19, 2016 at 9:27 PM, Jad Boutros  wrote:


We - at Snap - were forwarded this thread just a few hours ago and are
investigating. Please email me should you still be looking for a contact
for Snapchat.

Thank you,
Jad

On Mon, Dec 19, 2016 at 9:18 PM, Laurent Dumont 
wrote:


If anything comes from this, I'd love to hear about it. As a student in
the field, this is the kind of stuff I live for! ;)

Pretty awesome to see the chain of events after seeing a post on the
[pool] list!

Laurent

On 12/19/2016 05:12 PM, Justin Paine via NANOG wrote:


replying off list.


Justin Paine
Head of Trust & Safety
Cloudflare Inc.
PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D


On Mon, Dec 19, 2016 at 1:49 PM, Dan Drown  wrote:


Quoting David :


On 2016-12-19 1:55 PM, Jan Tore Morken wrote:


On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:


I found devices doing lookups for all of these at the same time

{0,0.uk,0.us,asia,europe,north-america,south-america,oceania
,africa}.pool.ntp.org
and then it proceeds to use everything returned, which explains why
everyone is seeing an increase.



Thanks, David. That perfectly matches the list of servers used by
older versions of the ios-ntp library[1][2], which would point toward
some iPhone app being the source of the traffic.

[1]
https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0
c976dd4aaeed37e0564/ios-ntp-rez/ntp.hosts
[2]
https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9d
ec9da5183e283eff9f2/ios-ntp-lib/NetworkClock.m#L122

That would make sense - I see a lot of iCloud related lookups from

these
hosts as well.

Also, app.snapchat.com generally seems to follow just after the NTP
pool
DNS lookups. I don't have an iPhone to test that though.



Confirmed - starting up the iOS Snapchat app does a lookup to the
domains
you listed, and then sends NTP to every unique IP.  Around 35-60
different
IPs.

Anyone have a contact at Snapchat?











---
Peter Beckman  Internet Guy
beck...@angryox.com http://www.angryox.com/
---

Re: Recent NTP pool traffic increase

[Jad Boutros via NANOG]

Thank you Eduardo,

You beat me to it. Apple was quick to approve our release with the fix, we
should start seeing a decrease in such traffic.

On Tue, Dec 20, 2016 at 2:29 PM, Eduardo Schoedler 
wrote:

> 2016-12-20 16:58 GMT-02:00 Jad Boutros via NANOG :
> > Immediately after being notified that our latest iOS release was causing
> > problems with NTP traffic, we started working to disable the offending
> code
> > in v9.45. We submitted a new mobile release to the Apple App Store
> earlier
> > this morning for their review, which should disable these NTP requests.
> We
> > are hoping Apple will be able to review this release in time before the
> > holiday break, and we have stressed its urgency. When the release does
> get
> > approved, we should very quickly begin to see a decrease in NTP traffic
> > from our app as users start upgrading to the new release.
>
> Just received update to 9.45.2.0.
>
> --
> Eduardo Schoedler
>

Re: Recent NTP pool traffic increase

[Jad Boutros via NANOG]

Immediately after being notified that our latest iOS release was causing
problems with NTP traffic, we started working to disable the offending code
in v9.45. We submitted a new mobile release to the Apple App Store earlier
this morning for their review, which should disable these NTP requests. We
are hoping Apple will be able to review this release in time before the
holiday break, and we have stressed its urgency. When the release does get
approved, we should very quickly begin to see a decrease in NTP traffic
from our app as users start upgrading to the new release.

We deeply regret this situation, and we will post an update here once we
hear back from Apple. We are also open to any suggestions on how we can
help with the present traffic.

On Mon, Dec 19, 2016 at 9:27 PM, Jad Boutros  wrote:

> We - at Snap - were forwarded this thread just a few hours ago and are
> investigating. Please email me should you still be looking for a contact
> for Snapchat.
>
> Thank you,
> Jad
>
> On Mon, Dec 19, 2016 at 9:18 PM, Laurent Dumont 
> wrote:
>
>> If anything comes from this, I'd love to hear about it. As a student in
>> the field, this is the kind of stuff I live for! ;)
>>
>> Pretty awesome to see the chain of events after seeing a post on the
>> [pool] list!
>>
>> Laurent
>>
>> On 12/19/2016 05:12 PM, Justin Paine via NANOG wrote:
>>
>>> replying off list.
>>>
>>> 
>>> Justin Paine
>>> Head of Trust & Safety
>>> Cloudflare Inc.
>>> PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
>>>
>>>
>>> On Mon, Dec 19, 2016 at 1:49 PM, Dan Drown  wrote:
>>>
 Quoting David :

> On 2016-12-19 1:55 PM, Jan Tore Morken wrote:
>
>> On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
>>
>>> I found devices doing lookups for all of these at the same time
>>>
>>> {0,0.uk,0.us,asia,europe,north-america,south-america,oceania
>>> ,africa}.pool.ntp.org
>>> and then it proceeds to use everything returned, which explains why
>>> everyone is seeing an increase.
>>>
>>
>> Thanks, David. That perfectly matches the list of servers used by
>> older versions of the ios-ntp library[1][2], which would point toward
>> some iPhone app being the source of the traffic.
>>
>> [1]
>> https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0
>> c976dd4aaeed37e0564/ios-ntp-rez/ntp.hosts
>> [2]
>> https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9d
>> ec9da5183e283eff9f2/ios-ntp-lib/NetworkClock.m#L122
>>
>> That would make sense - I see a lot of iCloud related lookups from
> these
> hosts as well.
>
> Also, app.snapchat.com generally seems to follow just after the NTP
> pool
> DNS lookups. I don't have an iPhone to test that though.
>

 Confirmed - starting up the iOS Snapchat app does a lookup to the
 domains
 you listed, and then sends NTP to every unique IP.  Around 35-60
 different
 IPs.

 Anyone have a contact at Snapchat?

>>>
>>
>

Re: Recent NTP pool traffic increase

[Paul Gear]

On 20/12/16 15:18, Laurent Dumont wrote:
> If anything comes from this, I'd love to hear about it. As a student in
> the field, this is the kind of stuff I live for! ;)
> 
> Pretty awesome to see the chain of events after seeing a post on the
> [pool] list!


https://news.ntppool.org/2016/12/load/

https://community.ntppool.org/t/recent-ntp-pool-traffic-increase/18

Re: Recent NTP pool traffic increase

[Jad Boutros via NANOG]

We - at Snap - were forwarded this thread just a few hours ago and are
investigating. Please email me should you still be looking for a contact
for Snapchat.

Thank you,
Jad

On Mon, Dec 19, 2016 at 9:18 PM, Laurent Dumont 
wrote:

> If anything comes from this, I'd love to hear about it. As a student in
> the field, this is the kind of stuff I live for! ;)
>
> Pretty awesome to see the chain of events after seeing a post on the
> [pool] list!
>
> Laurent
>
> On 12/19/2016 05:12 PM, Justin Paine via NANOG wrote:
>
>> replying off list.
>>
>> 
>> Justin Paine
>> Head of Trust & Safety
>> Cloudflare Inc.
>> PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
>>
>>
>> On Mon, Dec 19, 2016 at 1:49 PM, Dan Drown  wrote:
>>
>>> Quoting David :
>>>
 On 2016-12-19 1:55 PM, Jan Tore Morken wrote:

> On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
>
>> I found devices doing lookups for all of these at the same time
>>
>> {0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa}.
>> pool.ntp.org
>> and then it proceeds to use everything returned, which explains why
>> everyone is seeing an increase.
>>
>
> Thanks, David. That perfectly matches the list of servers used by
> older versions of the ios-ntp library[1][2], which would point toward
> some iPhone app being the source of the traffic.
>
> [1]
> https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0
> c976dd4aaeed37e0564/ios-ntp-rez/ntp.hosts
> [2]
> https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9d
> ec9da5183e283eff9f2/ios-ntp-lib/NetworkClock.m#L122
>
> That would make sense - I see a lot of iCloud related lookups from
 these
 hosts as well.

 Also, app.snapchat.com generally seems to follow just after the NTP
 pool
 DNS lookups. I don't have an iPhone to test that though.

>>>
>>> Confirmed - starting up the iOS Snapchat app does a lookup to the domains
>>> you listed, and then sends NTP to every unique IP.  Around 35-60
>>> different
>>> IPs.
>>>
>>> Anyone have a contact at Snapchat?
>>>
>>
>

Re: Recent NTP pool traffic increase

[Royce Williams]

On Mon, Dec 19, 2016 at 12:49 PM, Dan Drown <dan-na...@drown.org> wrote:
> Quoting David <open...@shaw.ca>:
>>
>> On 2016-12-19 1:55 PM, Jan Tore Morken wrote:
>>>
>>> On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
>>>>
>>>> I found devices doing lookups for all of these at the same time
>>>>
>>>> {0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa}.pool.ntp.org
>>>> and then it proceeds to use everything returned, which explains why
>>>> everyone is seeing an increase.
>>>
>>>
>>> Thanks, David. That perfectly matches the list of servers used by
>>> older versions of the ios-ntp library[1][2], which would point toward
>>> some iPhone app being the source of the traffic.
>>>
>>> [1]
>>> https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0564/ios-ntp-rez/ntp.hosts
>>> [2]
>>> https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff9f2/ios-ntp-lib/NetworkClock.m#L122
>>>
>>
>> That would make sense - I see a lot of iCloud related lookups from these
>> hosts as well.
>>
>> Also, app.snapchat.com generally seems to follow just after the NTP pool
>> DNS lookups. I don't have an iPhone to test that though.
>
>
> Confirmed - starting up the iOS Snapchat app does a lookup to the domains
> you listed, and then sends NTP to every unique IP.  Around 35-60 different
> IPs.
>
> Anyone have a contact at Snapchat?

Looks like folks got in touch with them. Thanks!

https://community.ntppool.org/t/recent-ntp-pool-traffic-increase/18

Royce

Re: Recent NTP pool traffic increase

[Roland Dobbins]

On 20 Dec 2016, at 12:18, Laurent Dumont wrote:

> As a student in the field, this is the kind of stuff I live for! ;)




---
Roland Dobbins 

Re: Recent NTP pool traffic increase

[Laurent Dumont]

If anything comes from this, I'd love to hear about it. As a student in 
the field, this is the kind of stuff I live for! ;)


Pretty awesome to see the chain of events after seeing a post on the 
[pool] list!


Laurent

On 12/19/2016 05:12 PM, Justin Paine via NANOG wrote:

replying off list.


Justin Paine
Head of Trust & Safety
Cloudflare Inc.
PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D


On Mon, Dec 19, 2016 at 1:49 PM, Dan Drown  wrote:

Quoting David :

On 2016-12-19 1:55 PM, Jan Tore Morken wrote:

On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:

I found devices doing lookups for all of these at the same time

{0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa}.pool.ntp.org
and then it proceeds to use everything returned, which explains why
everyone is seeing an increase.


Thanks, David. That perfectly matches the list of servers used by
older versions of the ios-ntp library[1][2], which would point toward
some iPhone app being the source of the traffic.

[1]
https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0564/ios-ntp-rez/ntp.hosts
[2]
https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff9f2/ios-ntp-lib/NetworkClock.m#L122


That would make sense - I see a lot of iCloud related lookups from these
hosts as well.

Also, app.snapchat.com generally seems to follow just after the NTP pool
DNS lookups. I don't have an iPhone to test that though.


Confirmed - starting up the iOS Snapchat app does a lookup to the domains
you listed, and then sends NTP to every unique IP.  Around 35-60 different
IPs.

Anyone have a contact at Snapchat?

Re: Recent NTP pool traffic increase

[Jan Tore Morken]

On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
> I found devices doing lookups for all of these at the same time
> {0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa,europe}.pool.ntp.org
> and then it proceeds to use everything returned, which explains why
> everyone is seeing an increase.

Thanks, David. That perfectly matches the list of servers used by
older versions of the ios-ntp library[1][2], which would point toward
some iPhone app being the source of the traffic.

[1] 
https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0564/ios-ntp-rez/ntp.hosts
[2] 
https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff9f2/ios-ntp-lib/NetworkClock.m#L122

-- 
Jan Tore Morken

Re: Recent NTP pool traffic increase

[Justin Paine via NANOG]

replying off list.


Justin Paine
Head of Trust & Safety
Cloudflare Inc.
PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D


On Mon, Dec 19, 2016 at 1:49 PM, Dan Drown  wrote:
> Quoting David :
>>
>> On 2016-12-19 1:55 PM, Jan Tore Morken wrote:
>>>
>>> On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:

 I found devices doing lookups for all of these at the same time

 {0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa}.pool.ntp.org
 and then it proceeds to use everything returned, which explains why
 everyone is seeing an increase.
>>>
>>>
>>> Thanks, David. That perfectly matches the list of servers used by
>>> older versions of the ios-ntp library[1][2], which would point toward
>>> some iPhone app being the source of the traffic.
>>>
>>> [1]
>>> https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0564/ios-ntp-rez/ntp.hosts
>>> [2]
>>> https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff9f2/ios-ntp-lib/NetworkClock.m#L122
>>>
>>
>> That would make sense - I see a lot of iCloud related lookups from these
>> hosts as well.
>>
>> Also, app.snapchat.com generally seems to follow just after the NTP pool
>> DNS lookups. I don't have an iPhone to test that though.
>
>
> Confirmed - starting up the iOS Snapchat app does a lookup to the domains
> you listed, and then sends NTP to every unique IP.  Around 35-60 different
> IPs.
>
> Anyone have a contact at Snapchat?

Re: Recent NTP pool traffic increase

[Dan Drown]

Quoting David :

On 2016-12-19 1:55 PM, Jan Tore Morken wrote:

On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:

I found devices doing lookups for all of these at the same time
{0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa}.pool.ntp.org
and then it proceeds to use everything returned, which explains why
everyone is seeing an increase.


Thanks, David. That perfectly matches the list of servers used by
older versions of the ios-ntp library[1][2], which would point toward
some iPhone app being the source of the traffic.

[1]  
https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0564/ios-ntp-rez/ntp.hosts
[2]  
https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff9f2/ios-ntp-lib/NetworkClock.m#L122




That would make sense - I see a lot of iCloud related lookups from  
these hosts as well.


Also, app.snapchat.com generally seems to follow just after the NTP  
pool DNS lookups. I don't have an iPhone to test that though.


Confirmed - starting up the iOS Snapchat app does a lookup to the  
domains you listed, and then sends NTP to every unique IP.  Around  
35-60 different IPs.


Anyone have a contact at Snapchat?

Re: Recent NTP pool traffic increase

[Justin Paine via NANOG]

the new Mario app perhaps? :)


Justin Paine
Head of Trust & Safety
Cloudflare Inc.
PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D


On Mon, Dec 19, 2016 at 1:12 PM, David  wrote:
> On 2016-12-19 1:55 PM, Jan Tore Morken wrote:
>>
>> On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
>>>
>>> I found devices doing lookups for all of these at the same time
>>>
>>> {0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa,europe}.pool.ntp.org
>>> and then it proceeds to use everything returned, which explains why
>>> everyone is seeing an increase.
>>
>>
>> Thanks, David. That perfectly matches the list of servers used by
>> older versions of the ios-ntp library[1][2], which would point toward
>> some iPhone app being the source of the traffic.
>>
>> [1]
>> https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0564/ios-ntp-rez/ntp.hosts
>> [2]
>> https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff9f2/ios-ntp-lib/NetworkClock.m#L122
>>
>
> That would make sense - I see a lot of iCloud related lookups from these
> hosts as well.
>
> Also, app.snapchat.com generally seems to follow just after the NTP pool DNS
> lookups. I don't have an iPhone to test that though.
>
> Thanks,
>

Re: Recent NTP pool traffic increase

[David]

On 2016-12-19 1:55 PM, Jan Tore Morken wrote:

On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:

I found devices doing lookups for all of these at the same time
{0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa,europe}.pool.ntp.org
and then it proceeds to use everything returned, which explains why
everyone is seeing an increase.


Thanks, David. That perfectly matches the list of servers used by
older versions of the ios-ntp library[1][2], which would point toward
some iPhone app being the source of the traffic.

[1] 
https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0564/ios-ntp-rez/ntp.hosts
[2] 
https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff9f2/ios-ntp-lib/NetworkClock.m#L122



That would make sense - I see a lot of iCloud related lookups from these 
hosts as well.


Also, app.snapchat.com generally seems to follow just after the NTP pool 
DNS lookups. I don't have an iPhone to test that though.


Thanks,

Re: Recent NTP pool traffic increase

[Ask Bjørn Hansen]

> On Dec 15, 2016, at 14:45, Jose Gerardo Perales Soto 
>  wrote:
> 
> We've recently experienced a traffic increase on the NTP queries to NTP pool 
> project (pool.ntp.org) servers. One theory is that some service provider NTP 
> infraestructure failed approximately 2 days ago and traffic is now being 
> redirected to servers belonging to the NTP pool project.

Hi Jose,

It’s more widespread than a particular service provider, so it seems more 
likely it’s a software update for some “IoT” device or similar.

The increase in DNS queries was on the “non-vendor” names, so it’s difficult to 
know who it is without being on a local network with one of the bad device 

The increase in DNS queries is much smaller than the increase in NTP queries 
that are being seen, so it’s not just more clients, but badly behaving ones. :-(

https://status.ntppool.org/incidents/vps6y4mm0m69

If you have NTP servers that can be added to the pool. it’d be greatly 
appreciated.

http://www.pool.ntp.org/join.html


Ask

Re: Recent NTP pool traffic increase

[Dan Drown]

Quoting David :
I found devices doing lookups for all of these at the same time  
{0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa,europe}.pool.ntp.org and then it proceeds to use everything returned, which explains why everyone is seeing an  
increase.


I'm very interested to find out what devices these are.  This would  
explain why places like New Zealand are getting massive amounts of NTP  
traffic from North America.

Re: Recent NTP pool traffic increase

[Valdis . Kletnieks]

On Mon, 19 Dec 2016 12:52:59 -0700, David said:

>  From a source network point of view we see devices come online and hit
> ~35 unique NTP servers within a few seconds.

Am I the only one who read that and started wondering if some engineer writing
CPE code read a recommendation someplace to "query 3-5 different servers" and
managed to miss the "-"?



pgpLj_BNMzrsW.pgp
Description: PGP signature

Re: Recent NTP pool traffic increase (update)

[Denys Fedoryshchenko]

I'm not sure if this issue relevant to discussed topic, Tenda routers 
here for a while on market, and i think i noticed this issue just now,
because NTP servers they are using supposedly for healthcheck went down 
(or NTP owners blocked ISP's i support, due such routers).


At least after checking numerous users, i believe Tenda hardcoded those 
NTP IPs. What worsen issue, that in Lebanon several times per day, for 
example at 18pm - short electricity cutoff,
and majority of users routers will reboot and surely reconnect, so it 
will look like a countrywide spike in NTP traffic.


I checked for a 10min also this NTP ips in dns responses, none of 
thousands of users tried to resolve any name with them over any DNS 
server, so i conclude they are hardcoded somewhere in firmware.


Here is traffic of Tenda router after reconnecting (but not full 
powercycle, i dont have it in my hands). But as you can see, no DNS 
resolution attempts:


20:15:59.305739 PPPoE  [ses 0x1483] CHAP, Success (0x03), id 1, Msg 
S=XX M=Authentication succeeded
20:15:59.306100 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1, 
length 12
20:15:59.317840 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1, 
length 24
20:15:59.317841 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 1, length 
12
20:15:59.317867 PPPoE  [ses 0x1483] IPCP, Conf-Nack (0x03), id 1, length 
18
20:15:59.325253 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 2, 
length 24
20:15:59.325273 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 2, length 
24
20:15:59.335589 PPPoE  [ses 0x1483] IP 172.17.49.245.123 > 
133.100.9.2.123: NTPv3, Client, length 48
20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 > 
192.5.41.41.123: NTPv3, Client, length 48
20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 > 
192.5.41.40.123: NTPv3, Client, length 48



Here is example of Tenda traffic if it is unable to reach destination, 
it repeats request each 10 seconds endlessly, my guess they are using 
ntp to show

status of internet connection.
So, now that NTP servers getting quite significant DDoS such way.

19:57:52.162863 IP (tos 0x0, ttl 64, id 38515, offset 0, flags [none], 
proto UDP (17), length 76)

172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length 48
	Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), 
precision 0

Root Delay: 0.00, Root dispersion: 0.00, Reference-ID: (unspec)
  Reference Timestamp:  0.0
  Originator Timestamp: 0.0
  Receive Timestamp:0.0
  Transmit Timestamp:   3691177063.0 (2016/12/19 22:57:43)
Originator - Receive Timestamp:  0.0
	Originator - Transmit Timestamp: 3691177063.0 (2016/12/19 
22:57:43)
19:57:52.163277 IP (tos 0x0, ttl 64, id 38516, offset 0, flags [none], 
proto UDP (17), length 76)

172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length 48
	Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), 
precision 0

Root Delay: 0.00, Root dispersion: 0.00, Reference-ID: (unspec)
  Reference Timestamp:  0.0
  Originator Timestamp: 0.0
  Receive Timestamp:0.0
  Transmit Timestamp:   3691177063.0 (2016/12/19 22:57:43)
Originator - Receive Timestamp:  0.0
	Originator - Transmit Timestamp: 3691177063.0 (2016/12/19 
22:57:43)
19:57:52.164435 IP (tos 0x0, ttl 64, id 38517, offset 0, flags [none], 
proto UDP (17), length 76)

172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length 48
	Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), 
precision 0

Root Delay: 0.00, Root dispersion: 0.00, Reference-ID: (unspec)
  Reference Timestamp:  0.0
  Originator Timestamp: 0.0
  Receive Timestamp:0.0
  Transmit Timestamp:   3691177063.0 (2016/12/19 22:57:43)
Originator - Receive Timestamp:  0.0
	Originator - Transmit Timestamp: 3691177063.0 (2016/12/19 
22:57:43)
19:58:02.164781 IP (tos 0x0, ttl 64, id 38518, offset 0, flags [none], 
proto UDP (17), length 76)

172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length 48
	Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), 
precision 0

Root Delay: 0.00, Root dispersion: 0.00, Reference-ID: (unspec)
  Reference Timestamp:  0.0
  Originator Timestamp: 0.0
  Receive Timestamp:0.0
  Transmit Timestamp:   3691177073.0 (2016/12/19 22:57:53)
Originator - Receive Timestamp:  0.0
	Originator - Transmit Timestamp: 3691177073.0 (2016/12/19 
22:57:53)
19:58:02.164884 IP (tos 0x0, ttl 64, id 38519, offset 0, flags [none], 
proto UDP (17), length 76)

172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length 48
	Client, Leap indicator:  

Re: Recent NTP pool traffic increase

[David]

On 2016-12-19 12:52 PM, David wrote:

On 2016-12-19 11:29 AM, Laurent Dumont wrote:

I also have a similar experience with an increased load.

I'm running a pretty basic Linode VPS and I had to fine tune a few
things in order to deal with the increased traffic. I can clearly see a
date around the 14-15 where my traffic increases to 3-4 times the usual
amounts.


From a source network point of view we see devices come online and hit
~35 unique NTP servers within a few seconds.

I'll try to see if I can track down what type of devices they are.



I found devices doing lookups for all of these at the same time 
{0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa,europe}.pool.ntp.org 
and then it proceeds to use everything returned, which explains why 
everyone is seeing an increase.

Re: Recent NTP pool traffic increase

[David]

On 2016-12-19 11:29 AM, Laurent Dumont wrote:

I also have a similar experience with an increased load.

I'm running a pretty basic Linode VPS and I had to fine tune a few
things in order to deal with the increased traffic. I can clearly see a
date around the 14-15 where my traffic increases to 3-4 times the usual
amounts.


From a source network point of view we see devices come online and hit 
~35 unique NTP servers within a few seconds.


I'll try to see if I can track down what type of devices they are.



I did a quick dump and in 60 seconds I was hit by slightly over 190K IPs

http://i.imgur.com/mygYINk.png

Weird stuff

Laurent


On 12/17/2016 10:25 PM, Gary E. Miller wrote:

Yo All!

On Sat, 17 Dec 2016 17:54:55 -0800
"Gary E. Miller"  wrote:


# tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"

And I do indeed get odd results.  Some on my local network...

To follow up on my own post, so this can be promply laid to rest.

After some discussion at NTPsec.  It seems that chronyd takes a lot
of 'creative license' with RFC 5905 (NTPv4).  But it is not malicious,
just 'odd', and not new.

So, nothing see here, back to the hunt for the real cause of the new
NTP traffic.

RGDS
GARY
---

Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588

Re: Recent NTP pool traffic increase (update)

[Denys Fedoryshchenko]

Many sorry! Update, seems illiterate in english (worse than me, hehe) 
customer was not precise about model of router, while he reported issue.


I noticed now many customers using specific models of routers reported 
issues with internet connection.
Analyzing internet traffic, i noticed that this routers seems 
excessively requesting ntp from those ip addresses, and not trying 
others:


 > 192.5.41.40.123: NTPv3, Client, length 48
 > 192.5.41.41.123: NTPv3, Client, length 48
 > 133.100.9.2.123: NTPv3, Client, length 48

I'm asking customer to make photo of device, to retrieve model and 
revision, and checking other customers as well, if they are abusing same 
servers.
There is definitely pattern, that all of them are using just this 3 
hardcoded servers. Problem is that many customers are changing mac of 
router, so i cannot clearly

identify vendor by first mac nibbles.
He sent me 2 photos, one of them LB-Link (mac vendor lookup 20:f4:1b 
says Shenzhen Bilian electronic CO.,LTD), another is Tenda (c8:3a:35 is 
Tenda).

If it is necessary i can investigate further.


On 2016-12-19 20:33, Ca By wrote:
My WAG is that the one plus updated firmeware on that day and they 
baked in

the pool.

Complete WAG, but time and distributed sources including wireless 
networks



On Mon, Dec 19, 2016 at 10:30 AM Laurent Dumont 


wrote:


I also have a similar experience with an increased load.



I'm running a pretty basic Linode VPS and I had to fine tune a few

things in order to deal with the increased traffic. I can clearly see 
a


date around the 14-15 where my traffic increases to 3-4 times the 
usual


amounts.



I did a quick dump and in 60 seconds I was hit by slightly over 190K 
IPs




http://i.imgur.com/mygYINk.png



Weird stuff



Laurent





On 12/17/2016 10:25 PM, Gary E. Miller wrote:

> Yo All!

>

> On Sat, 17 Dec 2016 17:54:55 -0800

> "Gary E. Miller"  wrote:

>

>> # tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"

>>

>> And I do indeed get odd results.  Some on my local network...

> To follow up on my own post, so this can be promply laid to rest.

>

> After some discussion at NTPsec.  It seems that chronyd takes a lot

> of 'creative license' with RFC 5905 (NTPv4).  But it is not malicious,

> just 'odd', and not new.

>

> So, nothing see here, back to the hunt for the real cause of the new

> NTP traffic.

>

> RGDS

> GARY

>
---

> Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703

>   g...@rellim.com  Tel:+1 541 382 8588

Re: Recent NTP pool traffic increase

[Denys Fedoryshchenko]

I noticed now many customers using tp-links reported issues with 
internet connection.
Analyzing internet traffic, i noticed that tp-link seems excessively 
requesting ntp from those ip addresses, and not trying others:


 > 192.5.41.40.123: NTPv3, Client, length 48
 > 192.5.41.41.123: NTPv3, Client, length 48
 > 133.100.9.2.123: NTPv3, Client, length 48

I'm asking customer to make photo of device, to retrieve model and 
revision, and checking other customers as well, if they are abusing same 
servers.


On 2016-12-19 20:33, Ca By wrote:
My WAG is that the one plus updated firmeware on that day and they 
baked in

the pool.

Complete WAG, but time and distributed sources including wireless 
networks



On Mon, Dec 19, 2016 at 10:30 AM Laurent Dumont 


wrote:


I also have a similar experience with an increased load.



I'm running a pretty basic Linode VPS and I had to fine tune a few

things in order to deal with the increased traffic. I can clearly see 
a


date around the 14-15 where my traffic increases to 3-4 times the 
usual


amounts.



I did a quick dump and in 60 seconds I was hit by slightly over 190K 
IPs




http://i.imgur.com/mygYINk.png



Weird stuff



Laurent





On 12/17/2016 10:25 PM, Gary E. Miller wrote:

> Yo All!

>

> On Sat, 17 Dec 2016 17:54:55 -0800

> "Gary E. Miller"  wrote:

>

>> # tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"

>>

>> And I do indeed get odd results.  Some on my local network...

> To follow up on my own post, so this can be promply laid to rest.

>

> After some discussion at NTPsec.  It seems that chronyd takes a lot

> of 'creative license' with RFC 5905 (NTPv4).  But it is not malicious,

> just 'odd', and not new.

>

> So, nothing see here, back to the hunt for the real cause of the new

> NTP traffic.

>

> RGDS

> GARY

>
---

> Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703

>   g...@rellim.com  Tel:+1 541 382 8588

Re: Recent NTP pool traffic increase

[Ca By]

My WAG is that the one plus updated firmeware on that day and they baked in
the pool.

Complete WAG, but time and distributed sources including wireless networks


On Mon, Dec 19, 2016 at 10:30 AM Laurent Dumont 
wrote:

> I also have a similar experience with an increased load.
>
>
>
> I'm running a pretty basic Linode VPS and I had to fine tune a few
>
> things in order to deal with the increased traffic. I can clearly see a
>
> date around the 14-15 where my traffic increases to 3-4 times the usual
>
> amounts.
>
>
>
> I did a quick dump and in 60 seconds I was hit by slightly over 190K IPs
>
>
>
> http://i.imgur.com/mygYINk.png
>
>
>
> Weird stuff
>
>
>
> Laurent
>
>
>
>
>
> On 12/17/2016 10:25 PM, Gary E. Miller wrote:
>
> > Yo All!
>
> >
>
> > On Sat, 17 Dec 2016 17:54:55 -0800
>
> > "Gary E. Miller"  wrote:
>
> >
>
> >> # tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"
>
> >>
>
> >> And I do indeed get odd results.  Some on my local network...
>
> > To follow up on my own post, so this can be promply laid to rest.
>
> >
>
> > After some discussion at NTPsec.  It seems that chronyd takes a lot
>
> > of 'creative license' with RFC 5905 (NTPv4).  But it is not malicious,
>
> > just 'odd', and not new.
>
> >
>
> > So, nothing see here, back to the hunt for the real cause of the new
>
> > NTP traffic.
>
> >
>
> > RGDS
>
> > GARY
>
> >
> ---
>
> > Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
>
> >   g...@rellim.com  Tel:+1 541 382 8588
>
>
>
>

Re: Recent NTP pool traffic increase

[Laurent Dumont]

I also have a similar experience with an increased load.

I'm running a pretty basic Linode VPS and I had to fine tune a few 
things in order to deal with the increased traffic. I can clearly see a 
date around the 14-15 where my traffic increases to 3-4 times the usual 
amounts.


I did a quick dump and in 60 seconds I was hit by slightly over 190K IPs

http://i.imgur.com/mygYINk.png

Weird stuff

Laurent


On 12/17/2016 10:25 PM, Gary E. Miller wrote:

Yo All!

On Sat, 17 Dec 2016 17:54:55 -0800
"Gary E. Miller"  wrote:


# tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"

And I do indeed get odd results.  Some on my local network...

To follow up on my own post, so this can be promply laid to rest.

After some discussion at NTPsec.  It seems that chronyd takes a lot
of 'creative license' with RFC 5905 (NTPv4).  But it is not malicious,
just 'odd', and not new.

So, nothing see here, back to the hunt for the real cause of the new
NTP traffic.

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588

Re: Recent NTP pool traffic increase

[Gary E. Miller]

Yo All!

On Sat, 17 Dec 2016 17:54:55 -0800
"Gary E. Miller"  wrote:

> # tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"
> 
> And I do indeed get odd results.  Some on my local network...

To follow up on my own post, so this can be promply laid to rest.

After some discussion at NTPsec.  It seems that chronyd takes a lot
of 'creative license' with RFC 5905 (NTPv4).  But it is not malicious,
just 'odd', and not new.

So, nothing see here, back to the hunt for the real cause of the new
NTP traffic.

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588


pgpoC_Gjv9NIU.pgp
Description: OpenPGP digital signature

Re: Recent NTP pool traffic increase

[Gary E. Miller]

Yo All!

Someone on nanog was reporrting on the new NTP mystery.  He suggested
doing a dump similar to this:

# tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"

And I do indeed get odd results.  Some on my local network...

This is from a chronyd host to an ntpsec host.  I monitor them both
continuously and both seem to be keeping good time.

17:36:11.369329 IP (tos 0x0, ttl 64, id 21405, offset 0, flags [DF], proto UDP (
17), length 76)
204.17.205.7.50937 > 204.17.205.27.123: [udp sum ok] NTPv4, length 48
Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecifi
ed), poll 6 (64s), precision 32
Root Delay: 0.00, Root dispersion: 0.00, Reference-ID: (unspec)
  Reference Timestamp:  0.0
  Originator Timestamp: 3691013707.207257069 (2016/12/17 17:35:07)
  Receive Timestamp:276521666.321684728 (2044/11/11 10:02:42)
  Transmit Timestamp:   3684123061.899235956 (2016/09/29 00:31:01)
Originator - Receive Timestamp:  +880475255.114427658
Originator - Transmit Timestamp: -6890645.308021113

That 'Receive Timestamp' is strange.

Here is another one from the same chronyd host, to another ntpsec host:

17:36:23.395415 IP (tos 0x0, ttl 64, id 3599, offset 0, flags [DF], proto UDP (1
7), length 76)
204.17.205.7.33551 > 204.17.205.1.123: [udp sum ok] NTPv4, length 48
Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecifi
ed), poll 6 (64s), precision 32
Root Delay: 0.00, Root dispersion: 0.00, Reference-ID: (unspec)
  Reference Timestamp:  0.0
  Originator Timestamp: 3691013718.824150890 (2016/12/17 17:35:18)
  Receive Timestamp:1779216017.648483479 (2092/06/24 18:08:33)
  Transmit Timestamp:   1405803137.064633429 (2080/08/24 20:20:33)
Originator - Receive Timestamp:  -1911797701.175667410
Originator - Transmit Timestamp: +2009756714.240482539

Note both the 'Receive Timestamp' and 'Transmit Timestamp' are both strange.

All three hosts have GPS for local time.

Here is one from a laptop, running chrony, that has not GPS:

17:36:52.643814 IP (tos 0x0, ttl 64, id 24624, offset 0, flags [DF], proto UDP (
17), length 76)
204.17.205.21.41485 > 204.17.205.8.123: [udp sum ok] NTPv4, length 48
Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 6 (64s), pre
cision 32
Root Delay: 0.00, Root dispersion: 0.00, Reference-ID: (unspec)
  Reference Timestamp:  0.0
  Originator Timestamp: 3691013747.797479298 (2016/12/17 17:35:47)
  Receive Timestamp:317494016.811980062 (2046/02/28 15:15:12)
  Transmit Timestamp:   127487236.597620268 (2040/02/21 11:35:32)
Originator - Receive Timestamp:  +921447565.014500764
Originator - Transmit Timestamp: +731440784.800140969

I have only seen this oddity from chronyd hosts...



RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588


pgplIfyQ3qLqR.pgp
Description: OpenPGP digital signature

Re: Recent NTP pool traffic increase

[Andreas Ott]

Hi,
On Fri, Dec 16, 2016 at 04:44:04PM +0700, Roland Dobbins wrote:
> > Looking at the source IP distribution, does a significant proportion 
> > of the larger query base seem to originate out-of-region?
> 
> And are do they appear to be mostly broadband access networks, or . . . 
> ?

Datapoints are via nfsen (nflow/sflow collection) from a US west coast
network lab that has "three" NTP pool servers, one IPv4 only set to 25
Mbps, the other one IPv4 and IPv6 on the same server both set to 100Mbps
at the NTP pool registration site.

Traffic is about 4 times P95 in the last 3 days from what it was before, and
the increase is IPv4 on the server that has IPv4 and IPv6. IPv6 traffic is
in line with what it used to be, no large increase.

The server with higher bandwidth and IPv4+IPv6 is seeing a large increase
on IPv4, from single hosts that seem to be in broadband networks and a certain
site's crawler that is hosted on AWS. The latter almost looks like someone
hardcoded a config instead of relying on the pool's DNS. 

The top talker abuses something in the protocol, this does not look for real and
I will contact Verizon/FiOS

tcpdump -nvvi hme0 port 123 and host 98.113.213.d|grep "Originator - Transmit 
Timestamp:"
Originator - Transmit Timestamp: 2123062516.816546608 (1967/04/12 
11:35:16)
Originator - Transmit Timestamp: 862276608.564645656 (1927/04/30 
01:16:48)
Originator - Transmit Timestamp: 3399899220.431115995 (2007/09/27 
16:27:00)
Originator - Transmit Timestamp: 140873162.935483905 (1904/06/19 
11:26:02)
Originator - Transmit Timestamp: 1878223676.912769495 (1959/07/09 
16:47:56)
Originator - Transmit Timestamp: 2713286246.929585296 (1985/12/24 
18:37:26)
Originator - Transmit Timestamp: 3219464534.831489402 (2002/01/08 
07:42:14)
Originator - Transmit Timestamp: 2210689093.339715993 (1970/01/20 
16:18:13)
Originator - Transmit Timestamp: 3899283084.650125848 (2023/07/25 
14:11:24)
[...]


nfdump -M /var/nfsen/profiles-data/live/dmz208_0201:br1  -T  -R 
2016/12/13/nfcapd.201612131630:2016/12/16/nfcapd.201612161630 -n 10 -s 
record/bytes -A proto,srcip,dstport -6 "dst ip j.k.l.235 and proto udp"
Aggregated flows 51346
Top 10 flows ordered by bytes:
Date first seen  Duration  Proto Src IP 
Addr Dst Pt   PacketsBytes  bpsBpp Flows
2016-12-13 16:31:22.608 259394.340  UDP 
98.113.213.d12312.3 M1.1 G34107 90  3000
2016-12-13 16:50:31.649 253960.650  UDP   
54.236.1.d123126976   11.4 M  359 9031
2016-12-13 17:43:29.760 255090.188  UDP   
54.236.1.d123114688   10.3 M  323 9028
2016-12-13 20:23:39.198 211054.259  UDP   
54.236.1.d123 901128.1 M  307 9022
2016-12-13 22:29:12.265 218623.774  UDP   204.177.184.d 
   123 614405.5 M  202 9015
2016-12-14 04:12:44.389 102634.717  UDP
162.243.191.d123 614405.5 M  431 9015
2016-12-13 22:10:33.226 223641.048  UDP 
198.199.99.d123 532484.8 M  171 9013
2016-12-13 21:31:18.841 194915.427  UDP   220.253.150.d 
   123 532484.8 M  196 9013
2016-12-13 20:01:40.452 242771.757  UDP  
troublemaker123 491524.4 M  145 9012
2016-12-14 05:21:20.634 208902.664  UDP   
54.236.1.d123 409603.7 M  141 9010
Summary: total flows: 60396, total bytes: 21023451720, total packets: 
233586118, avg bps: 648125, avg pps: 900, avg bpp: 90
Time window: 1970-01-01 00:00:01 - 2016-12-16 16:34:54
Total flows processed: 29676807, Blocks skipped: 0, Bytes read: 1662858132
Sys: 7.730s flows/second: 3839128.8  Wall: 7.722s flows/second: 3842810.0 

Note: "troublemaker" is a host on the internal network that has a known issue
with NTP time keeping, it originates a lot of packets and steps a lot.


Reply to me directly if you want more details.

-andreas
-- 
Andreas Ott   andr...@naund.org

Re: Recent NTP pool traffic increase

[Roland Dobbins]

On 16 Dec 2016, at 16:40, Roland Dobbins wrote:

Looking at the source IP distribution, does a significant proportion 
of the larger query base seem to originate out-of-region?


And are do they appear to be mostly broadband access networks, or . . . 
?


---
Roland Dobbins 

Re: Recent NTP pool traffic increase

[Roland Dobbins]

On 16 Dec 2016, at 6:20, Allan Liska wrote:

 FWIW The US server has seen a spike in traffic, but I have not seen a 
similar spike on the EMEA server.


Looking at the source IP distribution, does a significant proportion of 
the larger query base seem to originate out-of-region?


---
Roland Dobbins 

Re: Recent NTP pool traffic increase

[Allan Liska]

I manage two NTP servers in the pool, one in the US and the other in
EMEA.  FWIW The US server has seen a spike in traffic, but I have not
seen a similar spike on the EMEA server.  

allan

On 12/15/2016 at 5:46 PM, "Jose Gerardo Perales Soto"  wrote:Hi,

We've recently experienced a traffic increase on the NTP queries to
NTP pool project (pool.ntp.org) servers. One theory is that some
service provider NTP infraestructure failed approximately 2 days ago
and traffic is now being redirected to servers belonging to the NTP
pool project.

Does anyone from the service provider community have any comments?

Gerardo Perales

Re: Recent NTP pool traffic increase

[Roland Dobbins]

On 16 Dec 2016, at 10:17, Roland Dobbins wrote:





Over on nznog, Cameron Bradley posited that this may be related to a 
TR-069/-064 Mirai variant, which makes use of a 'SetNTPServers' exploit. 
 Perhaps one of them is actually setting timeservers?  This SANS 
writeup details the SOAP strings:




---
Roland Dobbins 

Re: Recent NTP pool traffic increase

[Roland Dobbins]

On 16 Dec 2016, at 10:16, Roland Dobbins wrote:

> 



---
Roland Dobbins 

Re: Recent NTP pool traffic increase

[Roland Dobbins]

On 16 Dec 2016, at 10:09, Dan Drown wrote:

 This seems more like "someone pushed out bad firmware" rather than 
something malicious.


Everything old is new again . . .



---
Roland Dobbins 

Re: Recent NTP pool traffic increase

[Dan Drown]

Quoting Roland Dobbins :
Do you have flow telemetry, which provides a lot more information  
than basic pps/bps stats?


Sources are pretty widely spread out among cell networks/home  
internet, seem to be mostly US based.  I'm not seeing a large amount  
of traffic per single IP or single subnet.  This seems more like  
"someone pushed out bad firmware" rather than something malicious.


Are you seeing normal timesync queries, or lots of level-6/level-7  
admin command attempts?


SNTP Client timesync queries make up 91.3% of the traffic to my server.

The following NTP settings being most the popular (47% of all traffic  
to my server):


stratum=0, poll=4, precision=-6, root delay=1, root dispersion=1,  
reference timestamp=0, originator timestamp=0,

receive timestamp=0

Re: Recent NTP pool traffic increase

[Roland Dobbins]

On 16 Dec 2016, at 5:45, Jose Gerardo Perales Soto wrote:

We've recently experienced a traffic increase on the NTP queries to 
NTP pool project (pool.ntp.org) servers.


Do you have flow telemetry, which provides a lot more information than 
basic pps/bps stats?


Are you seeing normal timesync queries, or lots of level-6/level-7 admin 
command attempts?


---
Roland Dobbins 

Re: Recent NTP pool traffic increase

[Kraig Beahn]

How much of a traffic increase?

On Dec 15, 2016 5:46 PM, "Jose Gerardo Perales Soto" <
gerardo.pera...@axtel.com.mx> wrote:

> Hi,
>
> We've recently experienced a traffic increase on the NTP queries to NTP
> pool project (pool.ntp.org) servers. One theory is that some service
> provider NTP infraestructure failed approximately 2 days ago and traffic is
> now being redirected to servers belonging to the NTP pool project.
>
> Does anyone from the service provider community have any comments?
>
> Gerardo Perales
>
>
> 
>
> NOTA: La información de este correo es de propiedad exclusiva y
> confidencial. Este mensaje es sólo para el destinatario señalado, si usted
> no lo es, destrúyalo de inmediato. Ninguna información aquí contenida debe
> ser entendida como dada o avalada por AXTEL, S.A.B. de C.V, sus
> subsidiarias o sus empleados, salvo cuando ello expresamente se indique. Es
> responsabilidad de quien recibe este correo de asegurarse que esté libre de
> virus, por lo tanto ni AXTEL, S.A.B. de C.V, sus subsidiarias ni sus
> empleados aceptan responsabilidad alguna.
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by AXTEL, S.A.B. de
> C.V, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither AXTEL, S.A.B. de C.V, its subsidiaries nor their
> employees accept any responsibility.
>

Re: Recent NTP pool traffic increase

[joel jaeggli]

On 12/15/16 3:07 PM, Dan Drown wrote:
> Quoting Jose Gerardo Perales Soto :
>> We've recently experienced a traffic increase on the NTP queries to
>> NTP pool project (pool.ntp.org) servers. One theory is that some
>> service provider NTP infraestructure failed approximately 2 days ago
>> and traffic is now being redirected to servers belonging to the NTP
>> pool project.
>>
>> Does anyone from the service provider community have any comments?
>
> To add some more numbers to this, I'm seeing 4x the usual NTP traffic
> to my server in pool.ntp.org, starting Dec 13.
>
> Top source ASNs by % of NTP traffic seen by my server (I don't have
> pre-Dec 13 traffic by ASN handy)
>
> sprint 4.0%
> verizon-wireless 3.4%
> tmobile 2.9%
> att-wireless 2.8%
> comcast 2.1%
> orange 1.8%
> sky 1.6%
> twc 1.0%
> att 1.0%
> swisscom 0.9%
> saudinet 0.8%
> virgin 0.6%
> opaltelecom 0.5%
> qwest 0.5%
> eli 0.2%
> verizon 0.2%
>
> Possibly related is the new iOS release.  Does the new iOS generate
> more NTP traffic?  Can anyone measure that?

IOS uses time.appple.com which is widely available.





signature.asc
Description: OpenPGP digital signature

Re: Recent NTP pool traffic increase

[Dan Drown]

Quoting Jose Gerardo Perales Soto :
We've recently experienced a traffic increase on the NTP queries to  
NTP pool project (pool.ntp.org) servers. One theory is that some  
service provider NTP infraestructure failed approximately 2 days ago  
and traffic is now being redirected to servers belonging to the NTP  
pool project.


Does anyone from the service provider community have any comments?


To add some more numbers to this, I'm seeing 4x the usual NTP traffic  
to my server in pool.ntp.org, starting Dec 13.


Top source ASNs by % of NTP traffic seen by my server (I don't have  
pre-Dec 13 traffic by ASN handy)


sprint 4.0%
verizon-wireless 3.4%
tmobile 2.9%
att-wireless 2.8%
comcast 2.1%
orange 1.8%
sky 1.6%
twc 1.0%
att 1.0%
swisscom 0.9%
saudinet 0.8%
virgin 0.6%
opaltelecom 0.5%
qwest 0.5%
eli 0.2%
verizon 0.2%

Possibly related is the new iOS release.  Does the new iOS generate  
more NTP traffic?  Can anyone measure that?

Re: Recent NTP pool traffic increase

[Blake Hudson]

I would think if a service provider failed, the stats would bear that 
out. For example, if one of the top ISPs in the world was forwarding 
requests, then you would likely see an increase in the number of queries 
generated from IP addresses registered to that organization. A similar 
effect could occur if a large ISP recently started distributing NTP 
servers as part of their DHCP options when they had not previously. If 
historical query data is not available, the current data could be used 
to make an educated guess and follow up on the likely data trails as 
currently visible.


I would also not rule out the possibility that a Netgear, DLink, 
T-mobile or some other vendor or distributor of access gear pushed out a 
firmware update which enabled NTP when it previously was disabled or 
otherwise changed a device's NTP settings or behavior.


--Blake

Jose Gerardo Perales Soto wrote on 12/15/2016 4:45 PM:

Hi,

We've recently experienced a traffic increase on the NTP queries to NTP pool 
project (pool.ntp.org) servers. One theory is that some service provider NTP 
infraestructure failed approximately 2 days ago and traffic is now being 
redirected to servers belonging to the NTP pool project.

Does anyone from the service provider community have any comments?

Gerardo Perales

Recent NTP pool traffic increase

[Jose Gerardo Perales Soto]

Hi,

We've recently experienced a traffic increase on the NTP queries to NTP pool 
project (pool.ntp.org) servers. One theory is that some service provider NTP 
infraestructure failed approximately 2 days ago and traffic is now being 
redirected to servers belonging to the NTP pool project.

Does anyone from the service provider community have any comments?

Gerardo Perales




NOTA: La información de este correo es de propiedad exclusiva y confidencial. 
Este mensaje es sólo para el destinatario señalado, si usted no lo es, 
destrúyalo de inmediato. Ninguna información aquí contenida debe ser entendida 
como dada o avalada por AXTEL, S.A.B. de C.V, sus subsidiarias o sus empleados, 
salvo cuando ello expresamente se indique. Es responsabilidad de quien recibe 
este correo de asegurarse que esté libre de virus, por lo tanto ni AXTEL, 
S.A.B. de C.V, sus subsidiarias ni sus empleados aceptan responsabilidad alguna.
NOTE: The information in this email is proprietary and confidential. This 
message is for the designated recipient only, if you are not the intended 
recipient, you should destroy it immediately. Any information in this message 
shall not be understood as given or endorsed by AXTEL, S.A.B. de C.V, its 
subsidiaries or their employees, unless expressly so stated. It is the 
responsibility of the recipient to ensure that this email is virus free, 
therefore neither AXTEL, S.A.B. de C.V, its subsidiaries nor their employees 
accept any responsibility.