Re: PlayStationNetwork blocking of CGNAT public addresses
On Thu, Sep 22, 2016 at 02:31:12PM +0200, Alexander Maassen wrote: > Maybe its time then for a global accepted, unified way to send/report abuse??? There are -- see Valdis's followup. But there's still no viable substitute for a working abuse@ address with clueful eyeballs on the other side of it. Every responsible and professional operation on this planet has that. The really good ones learn from what shows up there and pro-actively deal with abuse issues before anyone else is bothered by them, which not only makes them better netizens but reduces the volume of incoming complaints. ---rsk
Re: PlayStationNetwork blocking of CGNAT public addresses
On Mon, Sep 19, 2016 at 09:55:56PM +0200, Florian Weimer wrote: > Github users create several orders of magnitude more SSH connections > [snip] Ah. I didn't know that. Thanks! > Sure, and people already do this, and are not very flexible about it. > Support staff isn't briefed, and claim they do such stochastic > behavior adjustment across all (server) products, which I find > difficult to believe. You're right: those are serious drawbacks. If folks are going to do this, then they need to do it right, which means making sure everyone is in the loop and making sure that support staff are clueful/diligent enough to investigate -- or at least hand off to someone who'll investigate. This stuff works but only if you're adaptive/flexible and willing to learn and adjust on an ongoing basis. > I'm worried that this leads to a future where tunnelling everything > over HTTP(S) is no longer sufficient. You have to make it look like a > web server or browser, too. Everything else risks triggering > automated countermeasures. And as someone who constantly beats the "Internet != web" drum, I second this. Marginalizing other protocols doesn't serve us well in short term (it breaks things) or the long term (it stifles innovation). ---rsk
Re: PlayStationNetwork blocking of CGNAT public addresses
Well yes – if you have the automation, that is great. Of course the format of whatever log they send you matters too. I’ve had abuse complaints in a past life where the abuse report was a screenshot from a checkpoint firewall with “Dear team, for your attention” in bright red in a large font. Personally I don’t trash abuse reports that are valid. --srs From: Tom Beecher <beec...@beecher.cc> Date: Thursday, 22 September 2016 at 7:35 PM To: Brian Rak <b...@gameservers.com> Cc: Suresh Ramasubramanian <ops.li...@gmail.com>, "nanog@nanog.org" <nanog@nanog.org> Subject: Re: PlayStationNetwork blocking of CGNAT public addresses The format of the abuse complaint doesn't mean anything if it still doesn't contain any relevant data to say what the abuse IS. (Or, even if it IS abuse at all.) On Thu, Sep 22, 2016 at 9:37 AM, Brian Rak <b...@gameservers.com> wrote: Single IP per email: automated, zero time at all. Multiple IPs per email: manual process, minutes per IP. On 9/22/2016 9:34 AM, Suresh Ramasubramanian wrote: Considering that there are likely to be many such emails - just how much time is it going to take your abuse desk staffer to just parse out those IPs from whatever log that they send you? And how much time would processing say 50 individual emails take compared to 50 IPs in a single email? --srs On 22-Sep-2016, at 6:58 PM, Brian Rak <b...@gameservers.com <mailto:b...@gameservers.com>> wrote: We've also started ignoring their abuse emails, for the same reason. Their abuse emails at one point contained the line: > P.S. If you would prefer an individual email for each IP address on this > list, please let us know. But, they didn't respond after we contacted them requesting it (and that line has since been removed).
Re: PlayStationNetwork blocking of CGNAT public addresses
http://x-arf.org/ ? -- Hugo Slabbert | email, xmpp/jabber: h...@slabnet.com pgp key: B178313E | also on Signal On September 22, 2016 5:31:12 AM PDT, Alexander Maassenwrote: >Maybe its time then for a global accepted, unified way to send/report >abuse? >That should solve most of the issues and end points would be able to >deal with it in a common way and only would need to think about how to >integrate it in their crm's etc. >We are all using the same medium, but attempt to communicate issues >using several methods. >Perhaps iana can use those (m/b)illions they got from selling tld's and >cook something up. > > > >Kind regards, >Alexander Maassen >- Technical Maintenance Engineer Parkstad Support BV- Maintainer >DroneBL- Peplink Certified Engineer > pgpKk62VcDHj2.pgp Description: PGP signature
Re: PlayStationNetwork blocking of CGNAT public addresses
On Thu, 22 Sep 2016 14:31:12 +0200, Alexander Maassen said: > Maybe its time then for a global accepted, unified way to send/report abuse? YOu mean ike these RFCs? (OK, so it's an XML schema. Just be glad it isn't ASN.1 :) 5070 The Incident Object Description Exchange Format. R. Danyliw, J. Meijer, Y. Demchenko. December 2007. (Format: TXT=171529 bytes) (Updated by RFC6685) (Status: PROPOSED STANDARD) (DOI: 10.17487/RFC5070) 6684 Guidelines and Template for Defining Extensions to the Incident Object Description Exchange Format (IODEF). B. Trammell. July 2012. (Format: TXT=23550 bytes) (Status: INFORMATIONAL) (DOI: 10.17487/RFC6684) 6685 Expert Review for Incident Object Description Exchange Format (IODEF) Extensions in IANA XML Registry. B. Trammell. July 2012. (Format: TXT=4363 bytes) (Updates RFC5070) (Status: PROPOSED STANDARD) (DOI: 10.17487/RFC6685) 7203 An Incident Object Description Exchange Format (IODEF) Extension for Structured Cybersecurity Information. T. Takahashi, K. Landfield, Y. Kadobayashi. April 2014. (Format: TXT=57694 bytes) (Status: PROPOSED STANDARD) (DOI: 10.17487/RFC7203) 7495 Enumeration Reference Format for the Incident Object Description Exchange Format (IODEF). A. Montville, D. Black. March 2015. (Format: TXT=19891 bytes) (Status: PROPOSED STANDARD) (DOI: 10.17487/RFC7495) pgpY6BBIcDquc.pgp Description: PGP signature
Re: PlayStationNetwork blocking of CGNAT public addresses
The format of the abuse complaint doesn't mean anything if it still doesn't contain any relevant data to say what the abuse IS. (Or, even if it IS abuse at all.) On Thu, Sep 22, 2016 at 9:37 AM, Brian Rakwrote: > Single IP per email: automated, zero time at all. > > Multiple IPs per email: manual process, minutes per IP. > > > On 9/22/2016 9:34 AM, Suresh Ramasubramanian wrote: > >> Considering that there are likely to be many such emails - just how much >> time is it going to take your abuse desk staffer to just parse out those >> IPs from whatever log that they send you? >> >> And how much time would processing say 50 individual emails take compared >> to 50 IPs in a single email? >> >> --srs >> >> On 22-Sep-2016, at 6:58 PM, Brian Rak b...@gameservers.com>> wrote: >> >> We've also started ignoring their abuse emails, for the same reason. >>> Their abuse emails at one point contained the line: >>> >>> > P.S. If you would prefer an individual email for each IP address on >>> this list, please let us know. >>> >>> But, they didn't respond after we contacted them requesting it (and that >>> line has since been removed). >>> >> >
Re: PlayStationNetwork blocking of CGNAT public addresses
Single IP per email: automated, zero time at all. Multiple IPs per email: manual process, minutes per IP. On 9/22/2016 9:34 AM, Suresh Ramasubramanian wrote: Considering that there are likely to be many such emails - just how much time is it going to take your abuse desk staffer to just parse out those IPs from whatever log that they send you? And how much time would processing say 50 individual emails take compared to 50 IPs in a single email? --srs On 22-Sep-2016, at 6:58 PM, Brian Rak> wrote: We've also started ignoring their abuse emails, for the same reason. Their abuse emails at one point contained the line: > P.S. If you would prefer an individual email for each IP address on this list, please let us know. But, they didn't respond after we contacted them requesting it (and that line has since been removed).
Re: PlayStationNetwork blocking of CGNAT public addresses
Considering that there are likely to be many such emails - just how much time is it going to take your abuse desk staffer to just parse out those IPs from whatever log that they send you? And how much time would processing say 50 individual emails take compared to 50 IPs in a single email? --srs > On 22-Sep-2016, at 6:58 PM, Brian Rakwrote: > > We've also started ignoring their abuse emails, for the same reason. Their > abuse emails at one point contained the line: > > > P.S. If you would prefer an individual email for each IP address on this > > list, please let us know. > > But, they didn't respond after we contacted them requesting it (and that line > has since been removed).
Re: PlayStationNetwork blocking of CGNAT public addresses
On 9/22/2016 8:10 AM, Baldur Norddahl wrote: On 22 September 2016 at 10:42, Alexander Maassenwrote: So you ignore/don't deal with the abuse coz it's shipped in a format you refuse to handle? And you don't even bother telling the reporter you would like it in a per ip format? Or make attempts to make it work the way they report it (split out the ip's and modify the to be forwarded mail to only contain the ip's belonging to that customer) You will have to remember that these are automated mails from the reporter. If I write them back it goes into their bit bucket, because they do not really care enough to bother replying. I am betting they are sending out thousands mails each day and they can not handle manually replying to all of that. In the same way we receive a large amount of automated mail so we have to be able to handle it automatically. Send me something sane and I will make a script that forwards it. Send me something unusable and I wont - but I will not do manual handling of your automated mail. All I am trying to do here is tell people that send abuse mails not to combine multiple abuse complaints in one mail, because that makes it harder for everybody and makes it more likely that your mail will be dropped as too much work. Double so if your abuse mails is from an automated system, because I will try to match your automated system with my own. However it is much harder to make a system that can edit your complaint and duplicate it to several recipients, than it is to run a simple filter that just forwards the mail as is. As to PSN they will usually send multiple mails if the abuse is ongoing. At some point they will send a mail with just one IP and that one gets forwarded. So we are dropping some of the mails, but the users eventually get notified anyway. It is not ideal but it works. Regards, Baldur We've also started ignoring their abuse emails, for the same reason. Their abuse emails at one point contained the line: > P.S. If you would prefer an individual email for each IP address on this list, please let us know. But, they didn't respond after we contacted them requesting it (and that line has since been removed).
Re: PlayStationNetwork blocking of CGNAT public addresses
On Thursday, September 22, 2016, Alexander Maassen <outsi...@scarynet.org> wrote: > Both gamers and content providers do not care. The gamers as they only > care about the game itself and don't care about the technical mumbo jumbo. > And the makers coz they only care about making money by producing content > the gamers want. And you service providers are left with the headache of > attempts to please both sides. Very much agree > If this wasn't the case, then why after 20 years, ipv6 ain't rolled out. > Hence again I'd be voting for an ipv6 only day, but that will never > happen. Disagree. IPv6 is meaningfully rolled out. Half or comcast and at subs are observably on ipv6 http://www.worldipv6launch.org/measurements/ And every (i think) iphone 7 ships with ipv6 default on from t-mobile, sprint, T , and VZ. Same can be said of samsung phones 2 years ago. Now, if abc isp and xyz gaming company don't deploy ipv6, they have nobody to blame but themselves. Many of us have moved on, but it is sad when you all need help tweeking your cgn or need help finding an IPv4 broker. I feel your pain. But don't say ipv6 is not deployed. It is deployed, and it carries more traffic than ipv4 http://www.internetsociety.org/deploy360/blog/2016/08/facebook-akamai-pass-major-milestone-over-50-ipv6-from-us-mobile-networks/ CB Kind regards, > Alexander Maassen > - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- > Peplink Certified Engineer > > Oorspronkelijk bericht Van: Mark Andrews <ma...@isc.org > <javascript:;>> Datum: 21-09-16 03:29 (GMT+01:00) Aan: Justin Wilson < > li...@mtin.net <javascript:;>> Cc: NANOG <nanog@nanog.org <javascript:;>> > Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses > > In message <09342130-874f-4fa4-b410-b7b66a75f...@mtin.net <javascript:;>>, > Justin Wilson write > s: > > PSN is one reason I am not a fan of CGNAT. All they see are tons of > > connections from the same IP. This results in them banning folks. Due > > to them being hacked so many times getting them to actually communicate > > is almost impossible. My .02 is just get the gamers a true public if at > > all possible. > > > > Justin Wilson > > j...@mtin.net <javascript:;> > > What we need is business tech reporters to continually report on > these failures of content providers to deliver their services over > IPv6. 20 years lead time should be enough for any service. > > Mark > -- > Mark Andrews, ISC > 1 Seymour St., Dundas Valley, NSW 2117, Australia > PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org > <javascript:;> >
Re: PlayStationNetwork blocking of CGNAT public addresses
Ipv6 is there for 20+ years, cgnat is needed coz the net grows kinda exponentially due to stuff like IoT/mobiles/m2m, and isp's need to provide users with the ability to talk ipv4 simply because the other side refuses to deploy v6 abilities. Do the math if they really care. Also the servers itself hosting the gameserver probably already are dual stacked. But the gamecode itself misses the support. Then there is the issue of you as isp not being able or daring to show a fist and simply saying: screw you. Because you are risking to loose customers. And as long as the company's earn plenty of money using outdated code, they won't change it, coz that would imply spending money that won't flow into fancy buildings, fast cars and all that other useless luxury. Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer Oorspronkelijk bericht Van: Mike Hammett <na...@ics-il.net> Datum: 22-09-16 13:23 (GMT+01:00) Aan: Alexander Maassen <outsi...@scarynet.org> Cc: NANOG <nanog@nanog.org> Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses If you told them they would have fewer NAT issues if they supported IPv6, they'd start to care. ;-) They know enough to hate NAT. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP From: "Alexander Maassen" <outsi...@scarynet.org> Cc: "NANOG" <nanog@nanog.org> Sent: Thursday, September 22, 2016 3:35:01 AM Subject: Re: PlayStationNetwork blocking of CGNAT public addresses Both gamers and content providers do not care. The gamers as they only care about the game itself and don't care about the technical mumbo jumbo. And the makers coz they only care about making money by producing content the gamers want. And you service providers are left with the headache of attempts to please both sides. If this wasn't the case, then why after 20 years, ipv6 ain't rolled out. Hence again I'd be voting for an ipv6 only day, but that will never happen. Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer Oorspronkelijk bericht Van: Mark Andrews <ma...@isc.org> Datum: 21-09-16 03:29 (GMT+01:00) Aan: Justin Wilson <li...@mtin.net> Cc: NANOG <nanog@nanog.org> Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses In message <09342130-874f-4fa4-b410-b7b66a75f...@mtin.net>, Justin Wilson write s: > PSN is one reason I am not a fan of CGNAT. All they see are tons of > connections from the same IP. This results in them banning folks. Due > to them being hacked so many times getting them to actually communicate > is almost impossible. My .02 is just get the gamers a true public if at > all possible. > > Justin Wilson > j...@mtin.net What we need is business tech reporters to continually report on these failures of content providers to deliver their services over IPv6. 20 years lead time should be enough for any service. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: PlayStationNetwork blocking of CGNAT public addresses
Maybe its time then for a global accepted, unified way to send/report abuse? That should solve most of the issues and end points would be able to deal with it in a common way and only would need to think about how to integrate it in their crm's etc. We are all using the same medium, but attempt to communicate issues using several methods. Perhaps iana can use those (m/b)illions they got from selling tld's and cook something up. Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer Oorspronkelijk bericht Van: Baldur Norddahl <baldur.nordd...@gmail.com> Datum: 22-09-16 14:10 (GMT+01:00) Aan: nanog@nanog.org Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses On 22 September 2016 at 10:42, Alexander Maassen <outsi...@scarynet.org> wrote: > So you ignore/don't deal with the abuse coz it's shipped in a format you > refuse to handle? > > And you don't even bother telling the reporter you would like it in a per > ip format? Or make attempts to make it work the way they report it (split > out the ip's and modify the to be forwarded mail to only contain the ip's > belonging to that customer) > You will have to remember that these are automated mails from the reporter. If I write them back it goes into their bit bucket, because they do not really care enough to bother replying. I am betting they are sending out thousands mails each day and they can not handle manually replying to all of that. In the same way we receive a large amount of automated mail so we have to be able to handle it automatically. Send me something sane and I will make a script that forwards it. Send me something unusable and I wont - but I will not do manual handling of your automated mail. All I am trying to do here is tell people that send abuse mails not to combine multiple abuse complaints in one mail, because that makes it harder for everybody and makes it more likely that your mail will be dropped as too much work. Double so if your abuse mails is from an automated system, because I will try to match your automated system with my own. However it is much harder to make a system that can edit your complaint and duplicate it to several recipients, than it is to run a simple filter that just forwards the mail as is. As to PSN they will usually send multiple mails if the abuse is ongoing. At some point they will send a mail with just one IP and that one gets forwarded. So we are dropping some of the mails, but the users eventually get notified anyway. It is not ideal but it works. Regards, Baldur
Re: PlayStationNetwork blocking of CGNAT public addresses
On 22 September 2016 at 10:42, Alexander Maassenwrote: > So you ignore/don't deal with the abuse coz it's shipped in a format you > refuse to handle? > > And you don't even bother telling the reporter you would like it in a per > ip format? Or make attempts to make it work the way they report it (split > out the ip's and modify the to be forwarded mail to only contain the ip's > belonging to that customer) > You will have to remember that these are automated mails from the reporter. If I write them back it goes into their bit bucket, because they do not really care enough to bother replying. I am betting they are sending out thousands mails each day and they can not handle manually replying to all of that. In the same way we receive a large amount of automated mail so we have to be able to handle it automatically. Send me something sane and I will make a script that forwards it. Send me something unusable and I wont - but I will not do manual handling of your automated mail. All I am trying to do here is tell people that send abuse mails not to combine multiple abuse complaints in one mail, because that makes it harder for everybody and makes it more likely that your mail will be dropped as too much work. Double so if your abuse mails is from an automated system, because I will try to match your automated system with my own. However it is much harder to make a system that can edit your complaint and duplicate it to several recipients, than it is to run a simple filter that just forwards the mail as is. As to PSN they will usually send multiple mails if the abuse is ongoing. At some point they will send a mail with just one IP and that one gets forwarded. So we are dropping some of the mails, but the users eventually get notified anyway. It is not ideal but it works. Regards, Baldur
Re: PlayStationNetwork blocking of CGNAT public addresses
If you told them they would have fewer NAT issues if they supported IPv6, they'd start to care. ;-) They know enough to hate NAT. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Alexander Maassen" <outsi...@scarynet.org> Cc: "NANOG" <nanog@nanog.org> Sent: Thursday, September 22, 2016 3:35:01 AM Subject: Re: PlayStationNetwork blocking of CGNAT public addresses Both gamers and content providers do not care. The gamers as they only care about the game itself and don't care about the technical mumbo jumbo. And the makers coz they only care about making money by producing content the gamers want. And you service providers are left with the headache of attempts to please both sides. If this wasn't the case, then why after 20 years, ipv6 ain't rolled out. Hence again I'd be voting for an ipv6 only day, but that will never happen. Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer Oorspronkelijk bericht Van: Mark Andrews <ma...@isc.org> Datum: 21-09-16 03:29 (GMT+01:00) Aan: Justin Wilson <li...@mtin.net> Cc: NANOG <nanog@nanog.org> Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses In message <09342130-874f-4fa4-b410-b7b66a75f...@mtin.net>, Justin Wilson write s: > PSN is one reason I am not a fan of CGNAT. All they see are tons of > connections from the same IP. This results in them banning folks. Due > to them being hacked so many times getting them to actually communicate > is almost impossible. My .02 is just get the gamers a true public if at > all possible. > > Justin Wilson > j...@mtin.net What we need is business tech reporters to continually report on these failures of content providers to deliver their services over IPv6. 20 years lead time should be enough for any service. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: PlayStationNetwork blocking of CGNAT public addresses
As long as their is no international accepted standard as to how to report abuse and everyone cooking up his/her own methods.. I think you have either the choice of adapting and thus be able to deal with the abuse. Or be lazy and stubborn, ignore it, wait for the bad reputation to say hi to your company and face the effects it might cause. Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer Oorspronkelijk bericht Van: Tom Beecher <beec...@beecher.cc> Datum: 21-09-16 17:13 (GMT+01:00) Aan: Justin Wilson <li...@mtin.net> Cc: NANOG <nanog@nanog.org> Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses I have a hard time accepting that service providers should re-engineer their networks because other companies cannot properly engineer their abuse tooling. On Tue, Sep 20, 2016 at 11:33 AM, Justin Wilson <li...@mtin.net> wrote: > PSN is one reason I am not a fan of CGNAT. All they see are tons of > connections from the same IP. This results in them banning folks. Due to > them being hacked so many times getting them to actually communicate is > almost impossible. My .02 is just get the gamers a true public if at all > possible. > > Justin Wilson > j...@mtin.net > > --- > http://www.mtin.net Owner/CEO > xISP Solutions- Consulting – Data Centers - Bandwidth > > http://www.midwest-ix.com COO/Chairman > Internet Exchange - Peering - Distributed Fabric > > > On Sep 20, 2016, at 8:24 AM, Danijel Starman <theghost...@gmail.com> > wrote: > > > > Something similar happened to a local FantasyConon I was helping set up, > we > > had only two PS4 machines there and accounts provided by Blizzard for > > Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in > > about 8h. There was no other traffic other then those two accounts > playing > > Overwatch so my guess is that they have some too aggressive checks. I've > > managed to convince our ISP there to change the outside IP of the link so > > we got them working the next day but it happened again in 8h. > > > > -- > > *blap* > > > > On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart <si...@slimey.org> > wrote: > > > >> All, > >> > >> We operate an access network with several hundred thousand users. > >> Increasingly > >> we're putting the users behind CGNAT in order to continue to give them > an > >> IPv4 > >> service (we're all dual-stack, so they all get public IPv6 too). Due to > the > >> demographic of our users, many of them are gamers. > >> > >> We're hitting a problem with PlayStationNetwork 'randomly' blocking some > >> of our > >> CGNAT outside addresses, because they claim to have received anomalous, > or > >> 'attack' traffic from that IP. This obviously causes problems for the > other > >> legitimate users who end up behind the same public IPv4 address. > >> > >> Despite numerous attempts to engage with PSN, they are unwilling to > give us > >> any additional information which would allow us to identify the 'rogue' > >> users > >> on our network, or to identify the 'unwanted' traffic so that we could > >> either > >> block it, or use it to identify the rogue users ourselves. > >> > >> Has anyone else come up against the problem, and/or have any > suggestions on > >> how best to resolve it? > >> > >> Many thanks in advance, > >> > >> Simon > >> > >> > > > >
Re: PlayStationNetwork blocking of CGNAT public addresses
So you ignore/don't deal with the abuse coz it's shipped in a format you refuse to handle? And you don't even bother telling the reporter you would like it in a per ip format? Or make attempts to make it work the way they report it (split out the ip's and modify the to be forwarded mail to only contain the ip's belonging to that customer) Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer Oorspronkelijk bericht Van: Baldur Norddahl <baldur.nordd...@gmail.com> Datum: 21-09-16 10:37 (GMT+01:00) Aan: nanog@nanog.org Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses Hi We have the opposite problem with PSN: Sometimes they will send abuse reports with several of our IP addresses listed. The problem with that is that we can not give data about one customer to another customer. By listing multiple IP addresses we are prevented from forwarding the email to the customer. Which means we may ignore it instead. Regards, Baldur
Re: PlayStationNetwork blocking of CGNAT public addresses
Both gamers and content providers do not care. The gamers as they only care about the game itself and don't care about the technical mumbo jumbo. And the makers coz they only care about making money by producing content the gamers want. And you service providers are left with the headache of attempts to please both sides. If this wasn't the case, then why after 20 years, ipv6 ain't rolled out. Hence again I'd be voting for an ipv6 only day, but that will never happen. Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer Oorspronkelijk bericht Van: Mark Andrews <ma...@isc.org> Datum: 21-09-16 03:29 (GMT+01:00) Aan: Justin Wilson <li...@mtin.net> Cc: NANOG <nanog@nanog.org> Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses In message <09342130-874f-4fa4-b410-b7b66a75f...@mtin.net>, Justin Wilson write s: > PSN is one reason I am not a fan of CGNAT. All they see are tons of > connections from the same IP. This results in them banning folks. Due > to them being hacked so many times getting them to actually communicate > is almost impossible. My .02 is just get the gamers a true public if at > all possible. > > Justin Wilson > j...@mtin.net What we need is business tech reporters to continually report on these failures of content providers to deliver their services over IPv6. 20 years lead time should be enough for any service. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: PlayStationNetwork blocking of CGNAT public addresses
I have a hard time accepting that service providers should re-engineer their networks because other companies cannot properly engineer their abuse tooling. On Tue, Sep 20, 2016 at 11:33 AM, Justin Wilsonwrote: > PSN is one reason I am not a fan of CGNAT. All they see are tons of > connections from the same IP. This results in them banning folks. Due to > them being hacked so many times getting them to actually communicate is > almost impossible. My .02 is just get the gamers a true public if at all > possible. > > Justin Wilson > j...@mtin.net > > --- > http://www.mtin.net Owner/CEO > xISP Solutions- Consulting – Data Centers - Bandwidth > > http://www.midwest-ix.com COO/Chairman > Internet Exchange - Peering - Distributed Fabric > > > On Sep 20, 2016, at 8:24 AM, Danijel Starman > wrote: > > > > Something similar happened to a local FantasyConon I was helping set up, > we > > had only two PS4 machines there and accounts provided by Blizzard for > > Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in > > about 8h. There was no other traffic other then those two accounts > playing > > Overwatch so my guess is that they have some too aggressive checks. I've > > managed to convince our ISP there to change the outside IP of the link so > > we got them working the next day but it happened again in 8h. > > > > -- > > *blap* > > > > On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart > wrote: > > > >> All, > >> > >> We operate an access network with several hundred thousand users. > >> Increasingly > >> we're putting the users behind CGNAT in order to continue to give them > an > >> IPv4 > >> service (we're all dual-stack, so they all get public IPv6 too). Due to > the > >> demographic of our users, many of them are gamers. > >> > >> We're hitting a problem with PlayStationNetwork 'randomly' blocking some > >> of our > >> CGNAT outside addresses, because they claim to have received anomalous, > or > >> 'attack' traffic from that IP. This obviously causes problems for the > other > >> legitimate users who end up behind the same public IPv4 address. > >> > >> Despite numerous attempts to engage with PSN, they are unwilling to > give us > >> any additional information which would allow us to identify the 'rogue' > >> users > >> on our network, or to identify the 'unwanted' traffic so that we could > >> either > >> block it, or use it to identify the rogue users ourselves. > >> > >> Has anyone else come up against the problem, and/or have any > suggestions on > >> how best to resolve it? > >> > >> Many thanks in advance, > >> > >> Simon > >> > >> > > > >
Re: PlayStationNetwork blocking of CGNAT public addresses
On 21 Sep 2016, at 15:37, Baldur Norddahl wrote: Which means we may ignore it instead. . . . copy/paste or awk/sed or whatever isn't an option? If not, have you requested a) separate notifications per source and/or b) a more textual-manipulation-friendly format? Unless they're sending .gifs or something, surely this might be possible, yes? It seems within the realm of possibility this sort of response - or lack thereof - could result in some gaming network operators becoming a bit jaded. And perhaps some customers, too. --- Roland Dobbins
Re: PlayStationNetwork blocking of CGNAT public addresses
Hi We have the opposite problem with PSN: Sometimes they will send abuse reports with several of our IP addresses listed. The problem with that is that we can not give data about one customer to another customer. By listing multiple IP addresses we are prevented from forwarding the email to the customer. Which means we may ignore it instead. Regards, Baldur
Re: PlayStationNetwork blocking of CGNAT public addresses
On Wed, 21 Sep 2016 11:29:49 +1000, Mark Andrews said: > What we need is business tech reporters to continually report on > these failures of content providers to deliver their services over > IPv6. 20 years lead time should be enough for any service. Interestingly enough, the Playstation 4 has at least rudimentary IPv6 support - it will DHCPv6 and answer pings. Threw me for a loop first time I saw it, I couldn't figure out what unaccounted-for gear I had that was grabbing an IPv6 address... :) pgpXiDyor6qTh.pgp Description: PGP signature
Re: PlayStationNetwork blocking of CGNAT public addresses
Mark Andrews writes: > > In message <09342130-874f-4fa4-b410-b7b66a75f...@mtin.net>, Justin Wilson wri > te > s: > > PSN is one reason I am not a fan of CGNAT. All they see are tons of > > connections from the same IP. This results in them banning folks. Due > > to them being hacked so many times getting them to actually communicate > > is almost impossible. My .02 is just get the gamers a true public if at > > all possible. > > > > Justin Wilson > > j...@mtin.net > > What we need is business tech reporters to continually report on > these failures of content providers to deliver their services over > IPv6. 20 years lead time should be enough for any service. Additionally is the a role for the SEC in ensuring that companies take IPv6 seriously? If I remember correctly they got involved with Y2K. Just because there isn't a hard date it doesn't mean that IPv6 is any less important than Y2K to your business's survival. > Mark > -- > Mark Andrews, ISC > 1 Seymour St., Dundas Valley, NSW 2117, Australia > PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: PlayStationNetwork blocking of CGNAT public addresses
In message <09342130-874f-4fa4-b410-b7b66a75f...@mtin.net>, Justin Wilson write s: > PSN is one reason I am not a fan of CGNAT. All they see are tons of > connections from the same IP. This results in them banning folks. Due > to them being hacked so many times getting them to actually communicate > is almost impossible. My .02 is just get the gamers a true public if at > all possible. > > Justin Wilson > j...@mtin.net What we need is business tech reporters to continually report on these failures of content providers to deliver their services over IPv6. 20 years lead time should be enough for any service. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: PlayStationNetwork blocking of CGNAT public addresses
PSN is one reason I am not a fan of CGNAT. All they see are tons of connections from the same IP. This results in them banning folks. Due to them being hacked so many times getting them to actually communicate is almost impossible. My .02 is just get the gamers a true public if at all possible. Justin Wilson j...@mtin.net --- http://www.mtin.net Owner/CEO xISP Solutions- Consulting – Data Centers - Bandwidth http://www.midwest-ix.com COO/Chairman Internet Exchange - Peering - Distributed Fabric > On Sep 20, 2016, at 8:24 AM, Danijel Starmanwrote: > > Something similar happened to a local FantasyConon I was helping set up, we > had only two PS4 machines there and accounts provided by Blizzard for > Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in > about 8h. There was no other traffic other then those two accounts playing > Overwatch so my guess is that they have some too aggressive checks. I've > managed to convince our ISP there to change the outside IP of the link so > we got them working the next day but it happened again in 8h. > > -- > *blap* > > On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart wrote: > >> All, >> >> We operate an access network with several hundred thousand users. >> Increasingly >> we're putting the users behind CGNAT in order to continue to give them an >> IPv4 >> service (we're all dual-stack, so they all get public IPv6 too). Due to the >> demographic of our users, many of them are gamers. >> >> We're hitting a problem with PlayStationNetwork 'randomly' blocking some >> of our >> CGNAT outside addresses, because they claim to have received anomalous, or >> 'attack' traffic from that IP. This obviously causes problems for the other >> legitimate users who end up behind the same public IPv4 address. >> >> Despite numerous attempts to engage with PSN, they are unwilling to give us >> any additional information which would allow us to identify the 'rogue' >> users >> on our network, or to identify the 'unwanted' traffic so that we could >> either >> block it, or use it to identify the rogue users ourselves. >> >> Has anyone else come up against the problem, and/or have any suggestions on >> how best to resolve it? >> >> Many thanks in advance, >> >> Simon >> >> >
Re: PlayStationNetwork blocking of CGNAT public addresses
Something similar happened to a local FantasyConon I was helping set up, we had only two PS4 machines there and accounts provided by Blizzard for Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in about 8h. There was no other traffic other then those two accounts playing Overwatch so my guess is that they have some too aggressive checks. I've managed to convince our ISP there to change the outside IP of the link so we got them working the next day but it happened again in 8h. -- *blap* On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhartwrote: > All, > > We operate an access network with several hundred thousand users. > Increasingly > we're putting the users behind CGNAT in order to continue to give them an > IPv4 > service (we're all dual-stack, so they all get public IPv6 too). Due to the > demographic of our users, many of them are gamers. > > We're hitting a problem with PlayStationNetwork 'randomly' blocking some > of our > CGNAT outside addresses, because they claim to have received anomalous, or > 'attack' traffic from that IP. This obviously causes problems for the other > legitimate users who end up behind the same public IPv4 address. > > Despite numerous attempts to engage with PSN, they are unwilling to give us > any additional information which would allow us to identify the 'rogue' > users > on our network, or to identify the 'unwanted' traffic so that we could > either > block it, or use it to identify the rogue users ourselves. > > Has anyone else come up against the problem, and/or have any suggestions on > how best to resolve it? > > Many thanks in advance, > > Simon > >
Re: PlayStationNetwork blocking of CGNAT public addresses
* Rich Kulawiec: > On Sun, Sep 18, 2016 at 03:56:30PM +0200, Florian Weimer wrote: >> * Rich Kulawiec: >> >> > For example: if the average number of outbound SSH connections >> > established per hour per host across all hosts behind CGNAT is 3.2, >> > and you see a host making 1100/hour: that's a problem. It might be >> > someone who botched a Perl script; or it might be a botted host >> > trying to brute-force its way into something. >> >> If you do this, you break Github. > > 1. I didn't know that: *how* does this break Github? Github users create several orders of magnitude more SSH connections than average users because the most convenient way to set up read/write access is to use SSH. Depending on how you use Github, you might update lots and lots of local repositories from Github at certain times of the day. > 2. This is just an *example* of how to use the technique. It's not > meant to be literal. The general approach of determining the statistical > characteristics of "normal" and then flagging things that are "way > outside normal" works -- but of course it requires sufficient knowledge > to account for things like Github usage and/or infrequent events and/or > usage spikes triggered by real-world events, etc. Sure, and people already do this, and are not very flexible about it. Support staff isn't briefed, and claim they do such stochastic behavior adjustment across all (server) products, which I find difficult to believe. I'm worried that this leads to a future where tunnelling everything over HTTP(S) is no longer sufficient. You have to make it look like a web server or browser, too. Everything else risks triggering automated countermeasures. That's the anti-thesis of good protocol design.
Re: PlayStationNetwork blocking of CGNAT public addresses
On Sun, Sep 18, 2016 at 03:56:30PM +0200, Florian Weimer wrote: > * Rich Kulawiec: > > > For example: if the average number of outbound SSH connections > > established per hour per host across all hosts behind CGNAT is 3.2, > > and you see a host making 1100/hour: that's a problem. It might be > > someone who botched a Perl script; or it might be a botted host > > trying to brute-force its way into something. > > If you do this, you break Github. 1. I didn't know that: *how* does this break Github? 2. This is just an *example* of how to use the technique. It's not meant to be literal. The general approach of determining the statistical characteristics of "normal" and then flagging things that are "way outside normal" works -- but of course it requires sufficient knowledge to account for things like Github usage and/or infrequent events and/or usage spikes triggered by real-world events, etc. The more you do it, and the longer you do it, the better you'll get at it. (But of course the false positive rate will never be zero. That's why the question of what to do when anomalies happen isn't easy: poke a human? throttle? block? further analysis?) ---rsk
RE: PlayStationNetwork blocking of CGNAT public addresses
So the last one we successfully managed to isolate, our customer they had more than one PC with multiple infections. It’s not Playstation’s, but Windows machines that are infected with I assume some malware that is trying to log into PSN. cheers From: Jason Baugher [mailto:ja...@thebaughers.com] Sent: Monday, 19 September 2016 12:09 PM To: valdis.kletni...@vt.edu Cc: Tony Wicks <t...@wicks.co.nz>; NANOG <nanog@nanog.org> Subject: Re: PlayStationNetwork blocking of CGNAT public addresses So I should try again to get them to tell me what an "Account Takeover Attempt" is? They ignored my last request. It's easy to explain DMCA or spam to an end-user, but it's difficult to explain to some soccer mom that her kids are doing something to make Sony mad, when I can't explain to them what Sony is mad about. On Sun, Sep 18, 2016 at 5:58 PM, <valdis.kletni...@vt.edu <mailto:valdis.kletni...@vt.edu> > wrote: On Mon, 19 Sep 2016 10:41:59 +1200, "Tony Wicks" said: > Interestingly, Sony (SNEI-NOC-Abuse <SNEI-NOC-Abuse@am. > <mailto:SNEI-NOC-Abuse@am. %20 sony%20dot%20com)%20jut%0b>sony dot com) > jut > replied to being forwarded back one of their notification blocks requesting > more detailed information with a csv file in under an hour! So I guess name-and-shame *does* work? :)
Re: PlayStationNetwork blocking of CGNAT public addresses
On Mon, 19 Sep 2016 10:41:59 +1200, "Tony Wicks" said: > Interestingly, Sony (SNEI-NOC-Abusereplied to being forwarded back one of their notification blocks requesting > more detailed information with a csv file in under an hour! So I guess name-and-shame *does* work? :) pgp2syZkWt95D.pgp Description: PGP signature
RE: PlayStationNetwork blocking of CGNAT public addresses
Interestingly, Sony (SNEI-NOC-Abuse- Sony say no, either through silence, or explicitly.
Re: PlayStationNetwork blocking of CGNAT public addresses
On 9/18/2016 16:26, Larry Sheldon wrote: On 9/18/2016 08:19, Mike Hammett wrote: People love to hate incumbent telcos because of their arrogance (and frankly it's deserved), but people forget that big content can be just as arrogant and just as deserving of hatred. I never did see the benefit or the approach. To anybody. > I never did see the benefit oF the approach. To anybody. -- "Everybody is a genius. But if you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid." --Albert Einstein From Larry's Cox account.
Re: PlayStationNetwork blocking of CGNAT public addresses
On 9/18/2016 08:19, Mike Hammett wrote: People love to hate incumbent telcos because of their arrogance (and frankly it's deserved), but people forget that big content can be just as arrogant and just as deserving of hatred. I never did see the benefit or the approach. To anybody. -- "Everybody is a genius. But if you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid." --Albert Einstein From Larry's Cox account.
Re: PlayStationNetwork blocking of CGNAT public addresses
On Sun Sep 18, 2016 at 05:17:33PM +0200, Florian Weimer wrote: > Okay, then perhaps my guess of the ISP involved is wrong. It's not hard to find out who I work for :) > Out of curiosity, how common is end-to-end reporting of > source/destination port information (in addition to source IP > addresses and destination IP addresses)? Have the anti-abuse > mechanisms finalyl caught on with CGNAT, or is it possible that the > PSN operator themselves do not have such detailed data? 99.99% of abuse reports we receive contain the information, but that's because 99.99% of abuse reports we receive are from the 'copyright police', and their tools capture and include it in the reports. Once you discard that 99.99%, and are left with the stuff that is worthy of manual investigation, I'd say that almost all of it only contains timestamp and source IP. Sometimes it'll also contain destination IP (so we can take a best guess based on netflow data), and very occasionally it'll also contain source port information. I'd say the same also applies to requests for information that we receive from law enforcement agencies. In most cases, they're working from weblogs, and I'd be tempted to say that most webservers' 'out of the box' configuration does not log source port, only source IP in the web access logs. Simon
Re: PlayStationNetwork blocking of CGNAT public addresses
* Simon Lockhart: > On Sun Sep 18, 2016 at 03:58:57PM +0200, Florian Weimer wrote: >> * Tom Beecher: >> > Simon's getting screwed because he's not being given any information to try >> > and solve the problem, and because his customers are likely blaming him >> > because he's their ISP. >> >> We don't know that for sure. Another potential issue is that the ISP >> just cannot afford to notify its compromised customers, even if they >> were able to detect them. > > I'd like to think that we're pretty responsive to taking our users offline > when they're compromised and we're made aware of it - either through our own > tools, or through 3rd party notifications. Okay, then perhaps my guess of the ISP involved is wrong. > The process with Sony goes something like: > > - User reports they can't reach PSN > - We report the Sony/PSN, they say "Yes, it's blocked because that IP attacked > us" > - We say "Okay, that's a CGNAT public IP, can you help us identify the which > inside user that is - (timestamp,ip,port) logs, or some way to identify the > bad traffic so we can look for it ourselves" > - Sony say no, either through silence, or explicitly. > - We have unhappy user(s), who blame us. Yes, that's not very constructive. Out of curiosity, how common is end-to-end reporting of source/destination port information (in addition to source IP addresses and destination IP addresses)? Have the anti-abuse mechanisms finalyl caught on with CGNAT, or is it possible that the PSN operator themselves do not have such detailed data?
Re: PlayStationNetwork blocking of CGNAT public addresses
* Tom Beecher: > An email to a user notifying them they're likely compromised costs > basically nothing. If this increases the probability that the customer contacts customer support, in some markets, there is a risk that the account will never turn profitable during the current contract period. (Granted, my information may be woefully out of date, but my impression is that price-based competition is still pretty much cut-throat over here.) > If you find me an ISP that can't afford to notify users, I'll show > you one that shouldn't be in business anyways. I'm not blaming the ISP. (I may have done so in the past.) If we end up in such a situation, it's hardly the fault of one single ISP. > There's this presumption of guilt here, that Sony is right, and Simon's > subscribers are doing something malicious, yet they won't provide any > evidence of that. Even if they didn't know what it was, come back with > 'We're seeing weird bursts of [traffic characteristics] aimed at PSN during > these times. We're not quite sure what it is, but it's causing [problem > X].' It would still be a question of maliciousness or not, but it would be > something to work with. Providing nothing just perpetuates this finger > pointing game, and nothing gets solved. Yes, indeed. Resolving most networking problems needs cooperation, because at the most basic level, the Internet is still about connecting otherwise unrelated networks.
Re: PlayStationNetwork blocking of CGNAT public addresses
An email to a user notifying them they're likely compromised costs basically nothing. An email to their entire subscriber base also costs nothing. If you find me an ISP that can't afford to notify users, I'll show you one that shouldn't be in business anyways. There's this presumption of guilt here, that Sony is right, and Simon's subscribers are doing something malicious, yet they won't provide any evidence of that. Even if they didn't know what it was, come back with 'We're seeing weird bursts of [traffic characteristics] aimed at PSN during these times. We're not quite sure what it is, but it's causing [problem X].' It would still be a question of maliciousness or not, but it would be something to work with. Providing nothing just perpetuates this finger pointing game, and nothing gets solved. On Sun, Sep 18, 2016 at 9:58 AM, Florian Weimerwrote: > * Tom Beecher: > > > Simon's getting screwed because he's not being given any information to > try > > and solve the problem, and because his customers are likely blaming him > > because he's their ISP. > > We don't know that for sure. Another potential issue is that the ISP > just cannot afford to notify its compromised customers, even if they > were able to detect them. >
Re: PlayStationNetwork blocking of CGNAT public addresses
On Sun Sep 18, 2016 at 03:58:57PM +0200, Florian Weimer wrote: > * Tom Beecher: > > Simon's getting screwed because he's not being given any information to try > > and solve the problem, and because his customers are likely blaming him > > because he's their ISP. > > We don't know that for sure. Another potential issue is that the ISP > just cannot afford to notify its compromised customers, even if they > were able to detect them. I'd like to think that we're pretty responsive to taking our users offline when they're compromised and we're made aware of it - either through our own tools, or through 3rd party notifications. The process with Sony goes something like: - User reports they can't reach PSN - We report the Sony/PSN, they say "Yes, it's blocked because that IP attacked us" - We say "Okay, that's a CGNAT public IP, can you help us identify the which inside user that is - (timestamp,ip,port) logs, or some way to identify the bad traffic so we can look for it ourselves" - Sony say no, either through silence, or explicitly. - We have unhappy user(s), who blame us. Simon
Re: PlayStationNetwork blocking of CGNAT public addresses
* Tom Beecher: > Simon's getting screwed because he's not being given any information to try > and solve the problem, and because his customers are likely blaming him > because he's their ISP. We don't know that for sure. Another potential issue is that the ISP just cannot afford to notify its compromised customers, even if they were able to detect them.
Re: PlayStationNetwork blocking of CGNAT public addresses
* Rich Kulawiec: > For example: if the average number of outbound SSH connections > established per hour per host across all hosts behind CGNAT is 3.2, > and you see a host making 1100/hour: that's a problem. It might be > someone who botched a Perl script; or it might be a botted host > trying to brute-force its way into something. If you do this, you break Github. (If I guess Simon's network correctly, then I've seen reports which suggest that they might already be doing this.)
Re: PlayStationNetwork blocking of CGNAT public addresses
People love to hate incumbent telcos because of their arrogance (and frankly it's deserved), but people forget that big content can be just as arrogant and just as deserving of hatred. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Tom Beecher" <beec...@beecher.cc> To: "Tom Smyth" <tom.sm...@wirelessconnect.eu> Cc: "NANOG" <nanog@nanog.org> Sent: Sunday, September 18, 2016 8:15:08 AM Subject: Re: PlayStationNetwork blocking of CGNAT public addresses This is, as many things are, a huge problem in communication. Sony tells ISP 'Hey, you have customers abusing us. Fix it!'. ISP says 'Oh crap, sorry, what's going on? We'll run it down.' Sony says nothing. Let's just stop here for a second. This is fundamentally no different then the 'I have a problem, it's the network! complaints we've all dealt with forever. You spend days/weeks/months working on it. Maybe you ultimately find a goofy switchport, or maybe you discover that the server HDDs were crapping the bed and the problem server was chugging because of that. But you had to spend tons of time working on it because you couldn't get the info you need because the reporter was CONVINCED they KNEW what it was. Why should Simon have to spend hours of engineering time fishing through traffic captures and logs when he doesn't even know what he's LOOKING for? What does PSN consider 'abuse' here? Does Simon have customers infected with botnets that are targeting PSN at times? Or does PSN assume nobody will ever have more than a couple Playstations in a house, so if they see more than N connections to PSN from the same IP, it's malicious, since CGN is likely not something they considered? ( If anyone wants to place beer wagers, I'm picking the later. ) I spend about 8 weeks this year going back and forth with a Very Large Website Network who had blocked a /17 of IP space from accessing ANY of their sites because of 'malicious traffic' from a specific /23. 5 of those weeks, their responses consisted of 'it's malicious, you go find it, should be obvious', 'you clearly don't know what you're doing, we're wasting our time', etc. Week 5, I was able to extract that it was a specific web crawler that they said was knocking their databases over. After a conversation with their CIO the following week, they came back and admitted that a junior system admin made some PHP changes on a bunch of servers that he didn't think was in production,and when we crawled THOSE servers, Bad Things Happened for them. We were doing nothing wrong ; they just refused to look, and found it easier to blame us. Simon's getting screwed because he's not being given any information to try and solve the problem, and because his customers are likely blaming him because he's their ISP. Sony needs to stand up and work with him here. On Sun, Sep 18, 2016 at 8:30 AM, Tom Smyth <tom.sm...@wirelessconnect.eu> wrote: > Hi Simon, > > as other responders have said it is an inherent issue with NAT in general, > on workaround is to limit the ratio of actual users to an external IPv4 > address, the other thing we have seen from our Abuse contact emails from > PSN, is that malicious activity towards the PSN is often accompanied by > other malicious activities such as SSH brute force outbound and spaming... > > I would suggest that > > 1) limit the ratio of users to an external ipv4 address as much as possible > (which would reduce the impact of one compromised customer bringing down > play time for other clients behind the same nat > > 2)do some "canary in the mine" monitoring for obviously malicious traffic > (loads of SMTP traffic outbound) and lots of connection requests to SSH > servers ... if you see that traffic from behind your CGNAT device .. just > temporarily block the internal ip of the user until they clean up their > devices. > > this is the pain with NAT you have to do extra work in order prevent > infected users interrupting internet connectivity for other innocent > users... > I think you can use simple firewall rules on your edge router to identify > multiple connections to SMTP and SSH in a short period of time.. > > If you do the minimum to detect that abuse then you cant be accused of > invading peoples privacy... (bear in mind obvious false positives) > (Monitoring systems etc) ... > > Hope this helps, > > On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhart <si...@slimey.org> wrote: > > > All, > > > > We operate an access network with several hundred thousand users. > > Increasingly > > we're putting the users behind CGNAT in order to continue to give them an > > IPv4 > > service (we're all dual-st
Re: PlayStationNetwork blocking of CGNAT public addresses
This is, as many things are, a huge problem in communication. Sony tells ISP 'Hey, you have customers abusing us. Fix it!'. ISP says 'Oh crap, sorry, what's going on? We'll run it down.' Sony says nothing. Let's just stop here for a second. This is fundamentally no different then the 'I have a problem, it's the network! complaints we've all dealt with forever. You spend days/weeks/months working on it. Maybe you ultimately find a goofy switchport, or maybe you discover that the server HDDs were crapping the bed and the problem server was chugging because of that. But you had to spend tons of time working on it because you couldn't get the info you need because the reporter was CONVINCED they KNEW what it was. Why should Simon have to spend hours of engineering time fishing through traffic captures and logs when he doesn't even know what he's LOOKING for? What does PSN consider 'abuse' here? Does Simon have customers infected with botnets that are targeting PSN at times? Or does PSN assume nobody will ever have more than a couple Playstations in a house, so if they see more than N connections to PSN from the same IP, it's malicious, since CGN is likely not something they considered? ( If anyone wants to place beer wagers, I'm picking the later. ) I spend about 8 weeks this year going back and forth with a Very Large Website Network who had blocked a /17 of IP space from accessing ANY of their sites because of 'malicious traffic' from a specific /23. 5 of those weeks, their responses consisted of 'it's malicious, you go find it, should be obvious', 'you clearly don't know what you're doing, we're wasting our time', etc. Week 5, I was able to extract that it was a specific web crawler that they said was knocking their databases over. After a conversation with their CIO the following week, they came back and admitted that a junior system admin made some PHP changes on a bunch of servers that he didn't think was in production,and when we crawled THOSE servers, Bad Things Happened for them. We were doing nothing wrong ; they just refused to look, and found it easier to blame us. Simon's getting screwed because he's not being given any information to try and solve the problem, and because his customers are likely blaming him because he's their ISP. Sony needs to stand up and work with him here. On Sun, Sep 18, 2016 at 8:30 AM, Tom Smythwrote: > Hi Simon, > > as other responders have said it is an inherent issue with NAT in general, > on workaround is to limit the ratio of actual users to an external IPv4 > address, the other thing we have seen from our Abuse contact emails from > PSN, is that malicious activity towards the PSN is often accompanied by > other malicious activities such as SSH brute force outbound and spaming... > > I would suggest that > > 1) limit the ratio of users to an external ipv4 address as much as possible > (which would reduce the impact of one compromised customer bringing down > play time for other clients behind the same nat > > 2)do some "canary in the mine" monitoring for obviously malicious traffic > (loads of SMTP traffic outbound) and lots of connection requests to SSH > servers ... if you see that traffic from behind your CGNAT device .. just > temporarily block the internal ip of the user until they clean up their > devices. > > this is the pain with NAT you have to do extra work in order prevent > infected users interrupting internet connectivity for other innocent > users... > I think you can use simple firewall rules on your edge router to identify > multiple connections to SMTP and SSH in a short period of time.. > > If you do the minimum to detect that abuse then you cant be accused of > invading peoples privacy... (bear in mind obvious false positives) > (Monitoring systems etc) ... > > Hope this helps, > > On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhart wrote: > > > All, > > > > We operate an access network with several hundred thousand users. > > Increasingly > > we're putting the users behind CGNAT in order to continue to give them an > > IPv4 > > service (we're all dual-stack, so they all get public IPv6 too). Due to > the > > demographic of our users, many of them are gamers. > > > > We're hitting a problem with PlayStationNetwork 'randomly' blocking some > > of our > > CGNAT outside addresses, because they claim to have received anomalous, > or > > 'attack' traffic from that IP. This obviously causes problems for the > other > > legitimate users who end up behind the same public IPv4 address. > > > > Despite numerous attempts to engage with PSN, they are unwilling to give > us > > any additional information which would allow us to identify the 'rogue' > > users > > on our network, or to identify the 'unwanted' traffic so that we could > > either > > block it, or use it to identify the rogue users ourselves. > > > > Has anyone else come up against the problem, and/or have any suggestions > on > > how
Re: PlayStationNetwork blocking of CGNAT public addresses
On Sun, Sep 18, 2016 at 01:30:52PM +0100, Tom Smyth wrote: > 2)do some "canary in the mine" monitoring for obviously malicious traffic > (loads of SMTP traffic outbound) and lots of connection requests to SSH > servers ... if you see that traffic from behind your CGNAT device .. just > temporarily block the internal ip of the user until they clean up their > devices. Seconded. This is something I've recommended for years (decades, I suppose by now). Simple measurements of what's "normal" for your operation in terms of connection rates, types, etc., are easy to make. That in turn enables measurements of what's abnormal and that in turn enables manual or automatic actions. For example: if the average number of outbound SSH connections established per hour per host across all hosts behind CGNAT is 3.2, and you see a host making 1100/hour: that's a problem. It might be someone who botched a Perl script; or it might be a botted host trying to brute-force its way into something. These kinds of measurements are relatively easy to make and don't require invading user privacy. They won't catch everything, of course, but they're not intended to. They may catch enough to solve the problem in front of you at the moment *and*, if they do that, they may reduce the scope/scale of the rest of the problems to make them more tractable via other techniques. ---rsk
Re: PlayStationNetwork blocking of CGNAT public addresses
Hi Simon, as other responders have said it is an inherent issue with NAT in general, on workaround is to limit the ratio of actual users to an external IPv4 address, the other thing we have seen from our Abuse contact emails from PSN, is that malicious activity towards the PSN is often accompanied by other malicious activities such as SSH brute force outbound and spaming... I would suggest that 1) limit the ratio of users to an external ipv4 address as much as possible (which would reduce the impact of one compromised customer bringing down play time for other clients behind the same nat 2)do some "canary in the mine" monitoring for obviously malicious traffic (loads of SMTP traffic outbound) and lots of connection requests to SSH servers ... if you see that traffic from behind your CGNAT device .. just temporarily block the internal ip of the user until they clean up their devices. this is the pain with NAT you have to do extra work in order prevent infected users interrupting internet connectivity for other innocent users... I think you can use simple firewall rules on your edge router to identify multiple connections to SMTP and SSH in a short period of time.. If you do the minimum to detect that abuse then you cant be accused of invading peoples privacy... (bear in mind obvious false positives) (Monitoring systems etc) ... Hope this helps, On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhartwrote: > All, > > We operate an access network with several hundred thousand users. > Increasingly > we're putting the users behind CGNAT in order to continue to give them an > IPv4 > service (we're all dual-stack, so they all get public IPv6 too). Due to the > demographic of our users, many of them are gamers. > > We're hitting a problem with PlayStationNetwork 'randomly' blocking some > of our > CGNAT outside addresses, because they claim to have received anomalous, or > 'attack' traffic from that IP. This obviously causes problems for the other > legitimate users who end up behind the same public IPv4 address. > > Despite numerous attempts to engage with PSN, they are unwilling to give us > any additional information which would allow us to identify the 'rogue' > users > on our network, or to identify the 'unwanted' traffic so that we could > either > block it, or use it to identify the rogue users ourselves. > > Has anyone else come up against the problem, and/or have any suggestions on > how best to resolve it? > > Many thanks in advance, > > Simon > > -- Kindest regards, Tom Smyth Mobile: +353 87 6193172 - PLEASE CONSIDER THE ENVIRONMENT BEFORE YOU PRINT THIS E-MAIL This email contains information which may be confidential or privileged. The information is intended solely for the use of the individual or entity named above. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this electronic transmission in error, please notify me by telephone or by electronic mail immediately. Any opinions expressed are those of the author, not the company's .This email does not constitute either offer or acceptance of any contractually binding agreement. Such offer or acceptance must be communicated in writing. You are requested to carry out your own virus check before opening any attachment. Thomas Smyth accepts no liability for any loss or damage which may be caused by malicious software or attachments.
Re: PlayStationNetwork blocking of CGNAT public addresses
Simon Lockhart wrote: Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it? The best solution is to have a common practice on a set of public port numbers assigned to a host behind NAT. For example, with a practice that, if a port in a range between N*8 and N*8+7 is assigned to a host, other ports in the range is not assigned to other hosts, service providers can block packets based on IP addresses and ranges, especially if correspondence between hosts and ranges are rather stable. But, it may be too late to make such practice common, I'm afraid. Or, wait for a while until service providers receive enough amount of feedback from innocent users. To accelerate it, you can make correspondence between hosts and public addresses not so stable, which makes almost all your IP addresses marked bad quickly, which may make you loss some customer, unless other ISPs also do so. Masataka Ohta
RE: PlayStationNetwork blocking of CGNAT public addresses
Another aspect, for those users that need to go the PSN network but experience issues via the CGNAT, an opt-out solution (giving them public IPv4) may should mitigate the problem, that PSN network does not support IPv6. After all what percentage of your total subscribers that uses PSN and are gamers 2-3% ? Which might be relatively small amount to give public IPv4. Michalis -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Roland Dobbins Sent: Friday, September 16, 2016 4:32 PM To: nanog@nanog.org Subject: Re: PlayStationNetwork blocking of CGNAT public addresses On 16 Sep 2016, at 20:12, Simon Lockhart wrote: > Has anyone else come up against the problem, and/or have any > suggestions on how best to resolve it? I'm pretty sure that at least part of it has to do with DDoS-related activity. The best bet is to try and identify and engage with the relevant operational personnel with clue. Going the customer-service route isn't fruitful, as you indicate. Another aspect is ensuring that one has the ability to detect, classify, traceback, and mitigate outbound badness southbound of the CGN. This sort of thing has always been a problem with NAT; as CGN becomes more prevalent on wireline broadband networks, it's only going to get worse. AFAIK, PSN doesn't support IPv6. That would be another topic of discussion with the operational folks. --- Roland Dobbins <rdobb...@arbor.net>
RE: PlayStationNetwork blocking of CGNAT public addresses
So the pain has finally flowed down to other parts of the world. (APNIC ran out of IP's a long time ago, so CGN has been in use here for a lot longer) This issue is one I have been dealing with for the last four years. Only with Sony, no other company has caused such a headache in regard to CGNAT. I will not go into the long and painful saga of dealing with the constant issue of Sony putting blocks on random pool addresses, refusing to supply sufficient information to identify rouge users (timestamp, source IP, destination IP and port) then telling our customers it is a problem at the ISP end, but... Something happened about three months ago that Proves that if the Sony technical people want to get off their asses they are perfectly capable of supplying adequate information to identify a rogue user for the ISP to deal with. One of the local Sony PSN helpline managers actually managed to convince one of their technical people to supply a spreadsheet that magically contained sufficient information to allow us to identify a couple of users that did indeed have multiple infections. Great I thought, now if we can just get them to automate/regularly sent this info we will have a way forward. Alas, it appears it was a one off and we are back to the start. I will quote below what the Sony Network guy said when explaining why they can't send detailed information every time - " From: SNEI-NOC-Abuse [mailto:snei-noc-ab...@am.sony.com] Sent: Thursday, 11 August 2016 8:38 AM To: ##me## Cc: ##helpful Sony guy## Subject: RE: PSN / Flip Network blocks Hello, There is quite a bit of extra computing power required to produce the CSV file with timestamps and destination IP addresses. We send out over 6000 emails per day which already takes a significant amount of resources and time. We tend to get around 20-30 responses. Instead of wasting the resources on all those emails we generate CSV files for those who respond. We hope you understand. Thank you for taking action on these." So there you go, Sony can indeed solve this issue, but apparently a company that makes computers has insufficient computing power and staff to do so. Oh and after this, despite being asked many times they have never responded to requests for the CSV or similar detailed info. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Simon Lockhart Sent: Saturday, 17 September 2016 1:13 AM To: nanog@nanog.org Subject: PlayStationNetwork blocking of CGNAT public addresses All, We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers. We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address. Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves. Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it? Many thanks in advance, Simon
Re: PlayStationNetwork blocking of CGNAT public addresses
On Friday, September 16, 2016, Simon Lockhartwrote: > All, > > We operate an access network with several hundred thousand users. > Increasingly > we're putting the users behind CGNAT in order to continue to give them an > IPv4 > service (we're all dual-stack, so they all get public IPv6 too). Due to the > demographic of our users, many of them are gamers. > > We're hitting a problem with PlayStationNetwork 'randomly' blocking some > of our > CGNAT outside addresses, because they claim to have received anomalous, or > 'attack' traffic from that IP. This obviously causes problems for the other > legitimate users who end up behind the same public IPv4 address. > > Despite numerous attempts to engage with PSN, they are unwilling to give us > any additional information which would allow us to identify the 'rogue' > users > on our network, or to identify the 'unwanted' traffic so that we could > either > block it, or use it to identify the rogue users ourselves. > > Has anyone else come up against the problem, and/or have any suggestions on > how best to resolve it? > > Many thanks in advance, > > Simon Here is a picture of what you are experiencing http://test-ipv6.com/faq_avoids_ipv6.html Sometimes people need pictures to understand why IPv6 is important
Re: PlayStationNetwork blocking of CGNAT public addresses
Hi, as others have said, need to engage with one of their other units to get this sorted out - as a network provider, their customers are relying on YOU to access their service, PSN should care. technically, you could start looking at netflows to the PSN and see if anyone is engaged in DDoS via that route...and , if you offer IPv6 native service to end users, ask PSN when they are going to be offer an IPv6 service to their users - so this CGNAT stuff can go ;-) alan
Re: PlayStationNetwork blocking of CGNAT public addresses
On 16 Sep 2016, at 20:38, Simon Lockhart wrote: Unless we know what to look for, it's hard to detect and stop it. It's not just application-layer stuff - they're subject to all sorts of attacks. Screening out the obvious stuff would certainly help. The main issue is a dearth of engagement of clueful folks in the global operational community. Some gaming-oriented networks are well-represented; others are not, sadly. --- Roland Dobbins
Re: PlayStationNetwork blocking of CGNAT public addresses
On Fri Sep 16, 2016 at 08:32:12PM +0700, Roland Dobbins wrote: > Another aspect is ensuring that one has the ability to detect, classify, > traceback, and mitigate outbound badness southbound of the CGN. Unless PSN can tell us what traffic they consider bad, how can we detect and classify it? We certainly have the ability to traceback and mitigate, once we know what we're looking for. My understanding of the issue is that there are infected PCs on our network, which are being used as part of a distributed attack, but at the application layer, rather than network layer - distributed password brute-force, or similar. Unless we know what to look for, it's hard to detect and stop it. Simon
Re: PlayStationNetwork blocking of CGNAT public addresses
On 16 Sep 2016, at 20:12, Simon Lockhart wrote: Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it? I'm pretty sure that at least part of it has to do with DDoS-related activity. The best bet is to try and identify and engage with the relevant operational personnel with clue. Going the customer-service route isn't fruitful, as you indicate. Another aspect is ensuring that one has the ability to detect, classify, traceback, and mitigate outbound badness southbound of the CGN. This sort of thing has always been a problem with NAT; as CGN becomes more prevalent on wireline broadband networks, it's only going to get worse. AFAIK, PSN doesn't support IPv6. That would be another topic of discussion with the operational folks. --- Roland Dobbins
Re: PlayStationNetwork blocking of CGNAT public addresses
A network that doesn't support IPv6, yet discriminates against CGNAT? That seems like a promising future. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Simon Lockhart" <si...@slimey.org> To: nanog@nanog.org Sent: Friday, September 16, 2016 8:12:46 AM Subject: PlayStationNetwork blocking of CGNAT public addresses All, We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers. We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address. Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves. Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it? Many thanks in advance, Simon
PlayStationNetwork blocking of CGNAT public addresses
All, We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers. We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address. Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves. Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it? Many thanks in advance, Simon