Post mortem from Cybersource (and likely other payment gateways) on what happened on Tuesday...
"This was due to a network routing issue introduced by a third-party regional telecommunications service provider." *Incident Title* Transaction Failures via Multiple CyberSource Services *Status* Resolved *Incident Date/Duration* Start: 2018/07/17 16:48 (23:48 GMT) End: 2018/07/17 18:52 (01:52 GMT) Duration: 2 hours, 4 minutes. *Summary* CyberSource merchants experienced transaction failures utilizing multiple CyberSource service offerings dependent upon external gateways/processors. This was due to a network routing issue introduced by a third-party regional telecommunications service provider. CyberSource teams attempted to utilize the alternate datacenter and redundant network routes; however, this action failed to mitigate the issue. *Impact* During the incident period, merchants processing transactions via CyberSource datacenters received ESYSTEM and/or 150 error messages. Any transaction(s) that received these error messages failed processing. *Root Cause* A regional third-party telecommunications service provider introduced routing changes that affected communications between CyberSource and third-party gateways/processors. As this new routing information was more specific than the routing information advertised by CyberSource, reply information for requests initiated by CyberSource were not routed back as expected. As a result, CyberSource was unable to process these requests that were dependent on external processors/gateways. *Related and Contributing Factors* CyberSource monitoring systems immediately identified the issue at the application layer. As the impact emanated from an uncommunicated change instituted by the third-party telecommunications service, an increased amount of time was needed for determination of root cause and subsequent mitigation actions. This situation (lost return traffic) did not trigger alarms on the network components as they were functioning as designed. As no alerts are in place to monitor improper routing, there were no observed error conditions in any network device to pinpoint the issue for prompt remediation. *Resolution* CyberSource mitigated the incident by installing filters on CyberSource datacenter routers to ignore the third-party telecommunications route announcements. Once CyberSource installed the filters, expected transaction routing and processing resumed. *Action Taken for Remediation/Improvement* The following action(s) by CyberSource were taken and/or are planned to quickly identify and prevent an incident recurrence- CyberSource has implemented a change to ensure this particular third-party network is not able to cause this kind of issue in the future. CyberSource to include additional filtering on all existing and future traffic routes to prevent the inadvertent introduction of incorrect routing information. CyberSource is working with the third-party telecommunications network responsible for this impact to understand reasons behind this change, and identify process improvements to prevent a recurrence. ***Timeline and Actions Taken*** List of symptoms encountered and major actions taken by CyberSource during the incident. All times reported in Pacific Time (PST/PDT) and Greenwich Mean Time (GMT) in parenthesis. * Jun 17 16:48 (23:48) - Issue seen on outbound processor connection via CyberSource datacenters. Teams observe an application issue with respect to outbound traffic communication. Note: No network impact observed at this time. * 16:48 - 16:52 (23:48 - 23:52) - CyberSource fails transaction processing to alternate datacenter. Issue continues. * 17:06 (00:06) - CyberSource opens bridge for additional troubleshooting. * 17:49 (00:49) - CyberSource incorporates a routing filter to a single datacenter router. CyberSource observed stability processing through that device. * 18:52 (01:52)- Additional filters are installed on all edge devices. Errors on all processing platforms cease. Mike Castoro | Director of IT Adrianna Papell Group | 646-723-3885 From: Mike Castoro Sent: Tuesday, July 17, 2018 10:24 PM To: Phillip Lewis <phill...@joinvip.com> Cc: outages@outages.org Subject: Re: [outages] Cybersource Payment Gateway Thanks Phillip. Processing is back to normal for me as of 9:36pm ET. Mike Castoro Director of IT Adrianna Papell (646) 723-3885 mcast...@adriannapapell.com<mailto:mcast...@adriannapapell.com> ________________________________ From: Phillip Lewis <phill...@joinvip.com<mailto:phill...@joinvip.com>> Sent: Tuesday, July 17, 2018 9:05:41 PM To: Mike Castoro Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Cybersource Payment Gateway Confirmed. I got through to support. Major Processor outage affecting the card processors (not Cybersource directly). First Data and others involved. On Tue, Jul 17, 2018 at 9:02 PM Mike Castoro via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Anyone use Cybersource (a VISA Company) for credit card payment gateway and having issues? Been getting failed authorizations since 7:55PM EST. Been on hold with Support for a while. No alerts or notifications sent by Cybersource yet. Even tried to enter authorize directly in the virtual terminal. Reason Code 150 - General system failure The payment processor returned an unexpected result: Connection reset: gpn Apologies if this isn't the place for this type of issues (I saw a similar issue last week). Mike Castoro | Director of IT Adrianna Papell Group | New York, NY 10018 (646) 723-3885 | mcast...@adriannapapell.com<mailto:mcast...@adriannapapell.com> _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages -- Phillip Lewis Director of Operations VIP Communications Inc Office | (+1) 703-708-1515 Ext 3028 Email | phill...@joinvip.com<mailto:phill...@joinvip.com> Website | www.joinvip.com<http://www.joinvip.com/>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
