Re: Critical Denial of Service bugs in Discover
On Mon, Mar 7, 2022 at 1:16 PM Aleix Pol wrote: > > On Sat, Mar 5, 2022 at 8:36 AM Ben Cooksley wrote: > >> On Fri, Mar 4, 2022 at 12:49 AM Aleix Pol wrote: >> >>> I'd say wireshark is too low level for what the problem is here. We are >>> talking about having too many HTTP requests for specific URLs. >>> >> >> Correct, I guess the difference in our approaches comes from a "before >> release" to a "monitor after release" angle to things. >> I'd like to see increased scrutiny during the development process as well >> to make sure that we release code that operates properly from Day 1. >> > > A way to do this could be using commit hooks that do not allow to reach > certain services. (which we discussed in private chat). > We could also analyse at cmake time the knsrc files we install, but this > has a very limited and specific scope. > I've now applied two checks as part of the hooks which will hopefully catch anything new being introduced. We still need to ensure that anything pre-existing is sorted out of course. > > >> I can think two main measures: >>> - Trigger an alarm (an e-mail notification?) if there's a specific >>> UserAgent that has a specific portion of the queries we have in a specific >>> day in the services we care about. >>> - Offer plots to see how queries by UserAgent evolve over the last >>> couple of months (or couple of years). >>> >> >> At the moment our ability to analyse our logs is somewhat limited by our >> Privacy Policy - https://kde.org/privacypolicy/ >> Currently we don't have any provision for long term storage of >> this information even on an aggregated basis - so we would need to update >> this first. >> > > Hopefully the NDA should help here and it doesn't seem all that far away. > I know Neofytos and Ade have been working on it lately. > The privacy policy will still need to be updated, but that can form part of the puzzle yes. > > The second issue there is that we are transitioning users to contact a CDN >> based endpoint (which is substantially more scalable). >> This does mean we lose visibility on data such as User Agents and the >> URLs being impacted though as we only get aggregated data unless we ask for >> raw logs - which makes implementing something like what you've described >> much harder. >> > > That does seem like a stopper. Still, it seems like it's not that big of a > problem when there is a CDN, so we better worry about the other cases. > We should still be reasonable to the CDN of course, but it makes it much more managable yes. > > Aleix > Cheers, Ben
Re: Critical Denial of Service bugs in Discover
On Sat, Mar 5, 2022 at 8:36 AM Ben Cooksley wrote: > On Fri, Mar 4, 2022 at 12:49 AM Aleix Pol wrote: > >> I'd say wireshark is too low level for what the problem is here. We are >> talking about having too many HTTP requests for specific URLs. >> > > Correct, I guess the difference in our approaches comes from a "before > release" to a "monitor after release" angle to things. > I'd like to see increased scrutiny during the development process as well > to make sure that we release code that operates properly from Day 1. > A way to do this could be using commit hooks that do not allow to reach certain services. (which we discussed in private chat). We could also analyse at cmake time the knsrc files we install, but this has a very limited and specific scope. > I can think two main measures: >> - Trigger an alarm (an e-mail notification?) if there's a specific >> UserAgent that has a specific portion of the queries we have in a specific >> day in the services we care about. >> - Offer plots to see how queries by UserAgent evolve over the last couple >> of months (or couple of years). >> > > At the moment our ability to analyse our logs is somewhat limited by our > Privacy Policy - https://kde.org/privacypolicy/ > Currently we don't have any provision for long term storage of > this information even on an aggregated basis - so we would need to update > this first. > Hopefully the NDA should help here and it doesn't seem all that far away. I know Neofytos and Ade have been working on it lately. The second issue there is that we are transitioning users to contact a CDN > based endpoint (which is substantially more scalable). > This does mean we lose visibility on data such as User Agents and the URLs > being impacted though as we only get aggregated data unless we ask for > raw logs - which makes implementing something like what you've described > much harder. > That does seem like a stopper. Still, it seems like it's not that big of a problem when there is a CDN, so we better worry about the other cases. Aleix
Re: Critical Denial of Service bugs in Discover
On Fri, Mar 4, 2022 at 12:49 AM Aleix Pol wrote: > I'd say wireshark is too low level for what the problem is here. We are > talking about having too many HTTP requests for specific URLs. > Correct, I guess the difference in our approaches comes from a "before release" to a "monitor after release" angle to things. I'd like to see increased scrutiny during the development process as well to make sure that we release code that operates properly from Day 1. > > I can think two main measures: > - Trigger an alarm (an e-mail notification?) if there's a specific > UserAgent that has a specific portion of the queries we have in a specific > day in the services we care about. > - Offer plots to see how queries by UserAgent evolve over the last couple > of months (or couple of years). > At the moment our ability to analyse our logs is somewhat limited by our Privacy Policy - https://kde.org/privacypolicy/ Currently we don't have any provision for long term storage of this information even on an aggregated basis - so we would need to update this first. The second issue there is that we are transitioning users to contact a CDN based endpoint (which is substantially more scalable). This does mean we lose visibility on data such as User Agents and the URLs being impacted though as we only get aggregated data unless we ask for raw logs - which makes implementing something like what you've described much harder. > > Aleix > Cheers, Ben > > > On Thu, Mar 3, 2022 at 9:59 AM Ben Cooksley wrote: > >> On Thu, Mar 3, 2022 at 8:41 AM Aleix Pol wrote: >> >>> (dropping the distros list) >>> >>> @sysadmin have you been able to look into any tools we devs can have to >>> make sure this situation doesn't repeat in the future? >>> >> >> Hi Aleix, >> >> To be honest i've been struggling to think of ways that we could detect >> this on the server side prior to it becoming a massive issue. >> By the time an issue is evident server side it is usually much too late. >> >> The main tools i'd usually recommend would be the standard tools you >> would use for monitoring the network activity of any application - such as >> Wireshark. >> >> Is there something you were thinking of specifically in terms of us being >> able to provide? >> >> Thanks, >> Ben >> >> >>> >>> Aleix >>> >>> On Thu, Feb 10, 2022 at 1:10 PM Aleix Pol wrote: >>> On Thu, Feb 10, 2022 at 11:05 AM Ben Cooksley wrote: > > > > On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol wrote: >> >> [Snip] >> >> We still haven't discussed here is how to prevent this problem from >> happening again. >> >> If we don't have information about what is happening, we cannot fix problems. > > > Part of the issue here is that the problem only came to Sysadmin attention very recently, when the system ran out of disk space as a result of growing log files. > It was at that point we realised we had a serious problem. > > Prior to that the system load hadn't climbed to dangerous levels (> number of CPU cores) and Apache was keeping up with the traffic, so none of our other monitoring was tripped. > > If you have any thoughts on what sort of information you are thinking of that would be helpful. We could have plots of the amount of queries we get with a KNewStuff/* user-agent over time and their distribution. > It would definitely be helpful though to know when new software is going to be released that will be interacting with the servers as we will then be able to monitor for abnormalities. We make big announcements of every Plasma release... (?) >> Is there anything that could be done in this front? The issue here >> could have been addressed months ago, we just never knew it was >> happening. > > > One possibility that did occur to me today would be for us to integrate some kind of killswitch that our applications would check on first initialisation of functionality that talks to KDE.org servers. > This would allow us to disable the functionality in question on user systems. > > The check would only be done on first initialization to keep load low, while still ensuring all users eventually are affected by the killswitch (as they will eventually need to logout/reboot for some reason or another). > > The killswitch would probably work best if it had some kind of version check in it so we could specify which versions are disabled. > That would allow for subsequent updates - once delivered by distributions - to restore the functionality (while leaving it disabled for those who haven't updated). The file we are serving here effectively is the kill switch to all of KNewStuff. Aleix >>>
Re: Critical Denial of Service bugs in Discover
I'd say wireshark is too low level for what the problem is here. We are talking about having too many HTTP requests for specific URLs. I can think two main measures: - Trigger an alarm (an e-mail notification?) if there's a specific UserAgent that has a specific portion of the queries we have in a specific day in the services we care about. - Offer plots to see how queries by UserAgent evolve over the last couple of months (or couple of years). Aleix On Thu, Mar 3, 2022 at 9:59 AM Ben Cooksley wrote: > On Thu, Mar 3, 2022 at 8:41 AM Aleix Pol wrote: > >> (dropping the distros list) >> >> @sysadmin have you been able to look into any tools we devs can have to >> make sure this situation doesn't repeat in the future? >> > > Hi Aleix, > > To be honest i've been struggling to think of ways that we could detect > this on the server side prior to it becoming a massive issue. > By the time an issue is evident server side it is usually much too late. > > The main tools i'd usually recommend would be the standard tools you would > use for monitoring the network activity of any application - such as > Wireshark. > > Is there something you were thinking of specifically in terms of us being > able to provide? > > Thanks, > Ben > > >> >> Aleix >> >> On Thu, Feb 10, 2022 at 1:10 PM Aleix Pol wrote: >> >>> On Thu, Feb 10, 2022 at 11:05 AM Ben Cooksley wrote: >>> > >>> > >>> > >>> > On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol wrote: >>> >> >>> >> [Snip] >>> >> >>> >> We still haven't discussed here is how to prevent this problem from >>> >> happening again. >>> >> >>> >> If we don't have information about what is happening, we cannot fix >>> problems. >>> > >>> > >>> > Part of the issue here is that the problem only came to Sysadmin >>> attention very recently, when the system ran out of disk space as a result >>> of growing log files. >>> > It was at that point we realised we had a serious problem. >>> > >>> > Prior to that the system load hadn't climbed to dangerous levels (> >>> number of CPU cores) and Apache was keeping up with the traffic, so none of >>> our other monitoring was tripped. >>> > >>> > If you have any thoughts on what sort of information you are thinking >>> of that would be helpful. >>> >>> We could have plots of the amount of queries we get with a KNewStuff/* >>> user-agent over time and their distribution. >>> >>> > It would definitely be helpful though to know when new software is >>> going to be released that will be interacting with the servers as we will >>> then be able to monitor for abnormalities. >>> >>> We make big announcements of every Plasma release... (?) >>> >>> >> Is there anything that could be done in this front? The issue here >>> >> could have been addressed months ago, we just never knew it was >>> >> happening. >>> > >>> > >>> > One possibility that did occur to me today would be for us to >>> integrate some kind of killswitch that our applications would check on >>> first initialisation of functionality that talks to KDE.org servers. >>> > This would allow us to disable the functionality in question on user >>> systems. >>> > >>> > The check would only be done on first initialization to keep load low, >>> while still ensuring all users eventually are affected by the killswitch >>> (as they will eventually need to logout/reboot for some reason or another). >>> > >>> > The killswitch would probably work best if it had some kind of version >>> check in it so we could specify which versions are disabled. >>> > That would allow for subsequent updates - once delivered by >>> distributions - to restore the functionality (while leaving it disabled for >>> those who haven't updated). >>> >>> The file we are serving here effectively is the kill switch to all of >>> KNewStuff. >>> >>> Aleix >>> >>
Re: Critical Denial of Service bugs in Discover
On Thu, Mar 3, 2022 at 8:41 AM Aleix Pol wrote: > (dropping the distros list) > > @sysadmin have you been able to look into any tools we devs can have to > make sure this situation doesn't repeat in the future? > Hi Aleix, To be honest i've been struggling to think of ways that we could detect this on the server side prior to it becoming a massive issue. By the time an issue is evident server side it is usually much too late. The main tools i'd usually recommend would be the standard tools you would use for monitoring the network activity of any application - such as Wireshark. Is there something you were thinking of specifically in terms of us being able to provide? Thanks, Ben > > Aleix > > On Thu, Feb 10, 2022 at 1:10 PM Aleix Pol wrote: > >> On Thu, Feb 10, 2022 at 11:05 AM Ben Cooksley wrote: >> > >> > >> > >> > On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol wrote: >> >> >> >> [Snip] >> >> >> >> We still haven't discussed here is how to prevent this problem from >> >> happening again. >> >> >> >> If we don't have information about what is happening, we cannot fix >> problems. >> > >> > >> > Part of the issue here is that the problem only came to Sysadmin >> attention very recently, when the system ran out of disk space as a result >> of growing log files. >> > It was at that point we realised we had a serious problem. >> > >> > Prior to that the system load hadn't climbed to dangerous levels (> >> number of CPU cores) and Apache was keeping up with the traffic, so none of >> our other monitoring was tripped. >> > >> > If you have any thoughts on what sort of information you are thinking >> of that would be helpful. >> >> We could have plots of the amount of queries we get with a KNewStuff/* >> user-agent over time and their distribution. >> >> > It would definitely be helpful though to know when new software is >> going to be released that will be interacting with the servers as we will >> then be able to monitor for abnormalities. >> >> We make big announcements of every Plasma release... (?) >> >> >> Is there anything that could be done in this front? The issue here >> >> could have been addressed months ago, we just never knew it was >> >> happening. >> > >> > >> > One possibility that did occur to me today would be for us to integrate >> some kind of killswitch that our applications would check on first >> initialisation of functionality that talks to KDE.org servers. >> > This would allow us to disable the functionality in question on user >> systems. >> > >> > The check would only be done on first initialization to keep load low, >> while still ensuring all users eventually are affected by the killswitch >> (as they will eventually need to logout/reboot for some reason or another). >> > >> > The killswitch would probably work best if it had some kind of version >> check in it so we could specify which versions are disabled. >> > That would allow for subsequent updates - once delivered by >> distributions - to restore the functionality (while leaving it disabled for >> those who haven't updated). >> >> The file we are serving here effectively is the kill switch to all of >> KNewStuff. >> >> Aleix >> >
Re: Critical Denial of Service bugs in Discover
(dropping the distros list) @sysadmin have you been able to look into any tools we devs can have to make sure this situation doesn't repeat in the future? Aleix On Thu, Feb 10, 2022 at 1:10 PM Aleix Pol wrote: > On Thu, Feb 10, 2022 at 11:05 AM Ben Cooksley wrote: > > > > > > > > On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol wrote: > >> > >> [Snip] > >> > >> We still haven't discussed here is how to prevent this problem from > >> happening again. > >> > >> If we don't have information about what is happening, we cannot fix > problems. > > > > > > Part of the issue here is that the problem only came to Sysadmin > attention very recently, when the system ran out of disk space as a result > of growing log files. > > It was at that point we realised we had a serious problem. > > > > Prior to that the system load hadn't climbed to dangerous levels (> > number of CPU cores) and Apache was keeping up with the traffic, so none of > our other monitoring was tripped. > > > > If you have any thoughts on what sort of information you are thinking of > that would be helpful. > > We could have plots of the amount of queries we get with a KNewStuff/* > user-agent over time and their distribution. > > > It would definitely be helpful though to know when new software is going > to be released that will be interacting with the servers as we will then be > able to monitor for abnormalities. > > We make big announcements of every Plasma release... (?) > > >> Is there anything that could be done in this front? The issue here > >> could have been addressed months ago, we just never knew it was > >> happening. > > > > > > One possibility that did occur to me today would be for us to integrate > some kind of killswitch that our applications would check on first > initialisation of functionality that talks to KDE.org servers. > > This would allow us to disable the functionality in question on user > systems. > > > > The check would only be done on first initialization to keep load low, > while still ensuring all users eventually are affected by the killswitch > (as they will eventually need to logout/reboot for some reason or another). > > > > The killswitch would probably work best if it had some kind of version > check in it so we could specify which versions are disabled. > > That would allow for subsequent updates - once delivered by > distributions - to restore the functionality (while leaving it disabled for > those who haven't updated). > > The file we are serving here effectively is the kill switch to all of > KNewStuff. > > Aleix >
Re: Critical Denial of Service bugs in Discover
On Fri, Feb 25, 2022 at 10:09 PM Harald Sitter wrote: > On Mon, Feb 21, 2022 at 11:05 AM Ben Cooksley wrote: > > > > On Mon, Feb 21, 2022 at 10:01 PM Harald Sitter wrote: > >> > >> On Thu, Feb 10, 2022 at 1:11 PM Aleix Pol wrote: > >> > > >> > On Thu, Feb 10, 2022 at 11:05 AM Ben Cooksley > wrote: > >> > > > >> > > > >> > > > >> > > On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol wrote: > >> > >> > >> > >> [Snip] > >> > >> > >> > >> We still haven't discussed here is how to prevent this problem from > >> > >> happening again. > >> > >> > >> > >> If we don't have information about what is happening, we cannot > fix problems. > >> > > > >> > > > >> > > Part of the issue here is that the problem only came to Sysadmin > attention very recently, when the system ran out of disk space as a result > of growing log files. > >> > > It was at that point we realised we had a serious problem. > >> > > > >> > > Prior to that the system load hadn't climbed to dangerous levels (> > number of CPU cores) and Apache was keeping up with the traffic, so none of > our other monitoring was tripped. > >> > > > >> > > If you have any thoughts on what sort of information you are > thinking of that would be helpful. > >> > > >> > We could have plots of the amount of queries we get with a KNewStuff/* > >> > user-agent over time and their distribution. > >> > > >> > > It would definitely be helpful though to know when new software is > going to be released that will be interacting with the servers as we will > then be able to monitor for abnormalities. > >> > > >> > We make big announcements of every Plasma release... (?) > >> > > >> > >> Is there anything that could be done in this front? The issue here > >> > >> could have been addressed months ago, we just never knew it was > >> > >> happening. > >> > > > >> > > > >> > > One possibility that did occur to me today would be for us to > integrate some kind of killswitch that our applications would check on > first initialisation of functionality that talks to KDE.org servers. > >> > > This would allow us to disable the functionality in question on > user systems. > >> > > > >> > > The check would only be done on first initialization to keep load > low, while still ensuring all users eventually are affected by the > killswitch (as they will eventually need to logout/reboot for some reason > or another). > >> > > > >> > > The killswitch would probably work best if it had some kind of > version check in it so we could specify which versions are disabled. > >> > > That would allow for subsequent updates - once delivered by > distributions - to restore the functionality (while leaving it disabled for > those who haven't updated). > >> > > >> > The file we are serving here effectively is the kill switch to all of > KNewStuff. > >> > >> I'm a bit late to the party but for future reference I think this > >> was/is an architectural scaling problem on the server side as much as > >> a bug on the client. If just https load is the problem then the > >> "hotfix" is to use a HTTP load balancer until fixes make it into the > >> clients, killing the clients is like the last resort ever. I'm sure we > >> have the money to afford a bunch of cloud nodes serving as selective > >> proxy caches for a month to balance out the KNS load on the canonical > >> server. > > > > > > This was a multi-fold bug: > > > > 1) Sysadmin allowing a compatibility endpoint to remain alive for years > after we told people to stop using it and to use the new one (which is on a > CDN and which would have handled this whole issue much better) > > 2) Developers writing code to talk to KDE.org infrastructure without > consulting Sysadmin, especially where it deviated from previously > established patterns. > > > > In terms of scalability I disagree - the system is not being used here > in a manner for which it was not designed. > > > > This system is intended to serve downloads of KDE software and > associated data files to distributors and end users. These are actions that > are expected to: > > a) Be undertaken on an infrequent basis; and > > b) Be undertaken as a result of user initiated action (such as clicking > a download link) > > > > It was never intended to be used to serve configuration data files to > end user systems. We have autoconfig.kde.org for that. > > I'm not saying it should be. I'm saying instead of crippling our > software we should have made the architecture on the other end scale > as a hotfix and then fix the actual bug - that can still be hurried > along, I'm sure the involved developers would have been entirely > appreciative of the severity. The hip shooting doomsdaying I've seen > instead can surely only have demotivated people and came over as > unjustly aggressive and I'm sure peoples motivation to work on our > software in general has greatly suffered as a result. If we removed > features because they occasionally have bugs we may as well pack up > shop and stop making software altogether. > The '
Re: Critical Denial of Service bugs in Discover
On Mon, Feb 21, 2022 at 11:05 AM Ben Cooksley wrote: > > On Mon, Feb 21, 2022 at 10:01 PM Harald Sitter wrote: >> >> On Thu, Feb 10, 2022 at 1:11 PM Aleix Pol wrote: >> > >> > On Thu, Feb 10, 2022 at 11:05 AM Ben Cooksley wrote: >> > > >> > > >> > > >> > > On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol wrote: >> > >> >> > >> [Snip] >> > >> >> > >> We still haven't discussed here is how to prevent this problem from >> > >> happening again. >> > >> >> > >> If we don't have information about what is happening, we cannot fix >> > >> problems. >> > > >> > > >> > > Part of the issue here is that the problem only came to Sysadmin >> > > attention very recently, when the system ran out of disk space as a >> > > result of growing log files. >> > > It was at that point we realised we had a serious problem. >> > > >> > > Prior to that the system load hadn't climbed to dangerous levels (> >> > > number of CPU cores) and Apache was keeping up with the traffic, so none >> > > of our other monitoring was tripped. >> > > >> > > If you have any thoughts on what sort of information you are thinking of >> > > that would be helpful. >> > >> > We could have plots of the amount of queries we get with a KNewStuff/* >> > user-agent over time and their distribution. >> > >> > > It would definitely be helpful though to know when new software is going >> > > to be released that will be interacting with the servers as we will then >> > > be able to monitor for abnormalities. >> > >> > We make big announcements of every Plasma release... (?) >> > >> > >> Is there anything that could be done in this front? The issue here >> > >> could have been addressed months ago, we just never knew it was >> > >> happening. >> > > >> > > >> > > One possibility that did occur to me today would be for us to integrate >> > > some kind of killswitch that our applications would check on first >> > > initialisation of functionality that talks to KDE.org servers. >> > > This would allow us to disable the functionality in question on user >> > > systems. >> > > >> > > The check would only be done on first initialization to keep load low, >> > > while still ensuring all users eventually are affected by the killswitch >> > > (as they will eventually need to logout/reboot for some reason or >> > > another). >> > > >> > > The killswitch would probably work best if it had some kind of version >> > > check in it so we could specify which versions are disabled. >> > > That would allow for subsequent updates - once delivered by >> > > distributions - to restore the functionality (while leaving it disabled >> > > for those who haven't updated). >> > >> > The file we are serving here effectively is the kill switch to all of >> > KNewStuff. >> >> I'm a bit late to the party but for future reference I think this >> was/is an architectural scaling problem on the server side as much as >> a bug on the client. If just https load is the problem then the >> "hotfix" is to use a HTTP load balancer until fixes make it into the >> clients, killing the clients is like the last resort ever. I'm sure we >> have the money to afford a bunch of cloud nodes serving as selective >> proxy caches for a month to balance out the KNS load on the canonical >> server. > > > This was a multi-fold bug: > > 1) Sysadmin allowing a compatibility endpoint to remain alive for years after > we told people to stop using it and to use the new one (which is on a CDN and > which would have handled this whole issue much better) > 2) Developers writing code to talk to KDE.org infrastructure without > consulting Sysadmin, especially where it deviated from previously established > patterns. > > In terms of scalability I disagree - the system is not being used here in a > manner for which it was not designed. > > This system is intended to serve downloads of KDE software and associated > data files to distributors and end users. These are actions that are expected > to: > a) Be undertaken on an infrequent basis; and > b) Be undertaken as a result of user initiated action (such as clicking a > download link) > > It was never intended to be used to serve configuration data files to end > user systems. We have autoconfig.kde.org for that. I'm not saying it should be. I'm saying instead of crippling our software we should have made the architecture on the other end scale as a hotfix and then fix the actual bug - that can still be hurried along, I'm sure the involved developers would have been entirely appreciative of the severity. The hip shooting doomsdaying I've seen instead can surely only have demotivated people and came over as unjustly aggressive and I'm sure peoples motivation to work on our software in general has greatly suffered as a result. If we removed features because they occasionally have bugs we may as well pack up shop and stop making software altogether. > The system in question is handling the load extremely well and far beyond my > expectations - it is fairly un
Re: Critical Denial of Service bugs in Discover
On Mon, Feb 21, 2022 at 10:01 PM Harald Sitter wrote: > On Thu, Feb 10, 2022 at 1:11 PM Aleix Pol wrote: > > > > On Thu, Feb 10, 2022 at 11:05 AM Ben Cooksley wrote: > > > > > > > > > > > > On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol wrote: > > >> > > >> [Snip] > > >> > > >> We still haven't discussed here is how to prevent this problem from > > >> happening again. > > >> > > >> If we don't have information about what is happening, we cannot fix > problems. > > > > > > > > > Part of the issue here is that the problem only came to Sysadmin > attention very recently, when the system ran out of disk space as a result > of growing log files. > > > It was at that point we realised we had a serious problem. > > > > > > Prior to that the system load hadn't climbed to dangerous levels (> > number of CPU cores) and Apache was keeping up with the traffic, so none of > our other monitoring was tripped. > > > > > > If you have any thoughts on what sort of information you are thinking > of that would be helpful. > > > > We could have plots of the amount of queries we get with a KNewStuff/* > > user-agent over time and their distribution. > > > > > It would definitely be helpful though to know when new software is > going to be released that will be interacting with the servers as we will > then be able to monitor for abnormalities. > > > > We make big announcements of every Plasma release... (?) > > > > >> Is there anything that could be done in this front? The issue here > > >> could have been addressed months ago, we just never knew it was > > >> happening. > > > > > > > > > One possibility that did occur to me today would be for us to > integrate some kind of killswitch that our applications would check on > first initialisation of functionality that talks to KDE.org servers. > > > This would allow us to disable the functionality in question on user > systems. > > > > > > The check would only be done on first initialization to keep load low, > while still ensuring all users eventually are affected by the killswitch > (as they will eventually need to logout/reboot for some reason or another). > > > > > > The killswitch would probably work best if it had some kind of version > check in it so we could specify which versions are disabled. > > > That would allow for subsequent updates - once delivered by > distributions - to restore the functionality (while leaving it disabled for > those who haven't updated). > > > > The file we are serving here effectively is the kill switch to all of > KNewStuff. > > I'm a bit late to the party but for future reference I think this > was/is an architectural scaling problem on the server side as much as > a bug on the client. If just https load is the problem then the > "hotfix" is to use a HTTP load balancer until fixes make it into the > clients, killing the clients is like the last resort ever. I'm sure we > have the money to afford a bunch of cloud nodes serving as selective > proxy caches for a month to balance out the KNS load on the canonical > server. > This was a multi-fold bug: 1) Sysadmin allowing a compatibility endpoint to remain alive for years after we told people to stop using it and to use the new one (which is on a CDN and which would have handled this whole issue much better) 2) Developers writing code to talk to KDE.org infrastructure without consulting Sysadmin, especially where it deviated from previously established patterns. In terms of scalability I disagree - the system is not being used here in a manner for which it was not designed. This system is intended to serve downloads of KDE software and associated data files to distributors and end users. These are actions that are expected to: a) Be undertaken on an infrequent basis; and b) Be undertaken as a result of user initiated action (such as clicking a download link) It was never intended to be used to serve configuration data files to end user systems. We have autoconfig.kde.org for that. The system in question is handling the load extremely well and far beyond my expectations - it is fairly unfathomable that download.kde.org and files.kde.org would receive traffic on the order of 500-600 requests per second. During this time the highest load I have seen has been around 8 - and despite this being uncomfortably busy it has not fallen over or dropped the ball for both it's BAU activity as well as the abuse it has taken. (My extreme level of concern on this matter has been because I knew that if we hit a major release such as for Krita we would likely reach the point that the system would no longer be able to cope) I should note that I do not believe in temporary fixes for any issue, as they often become permanent fixes - and a cluster of cloud nodes would have to remain around for many months if not longer had I not aggressively pushed for the fixes to be backported. Case in point here is Plasma - several of the applications still using download.kde.org/ocs/providers.xml were in Plasma - being KWin
Re: Critical Denial of Service bugs in Discover
On Thu, Feb 10, 2022 at 1:11 PM Aleix Pol wrote: > > On Thu, Feb 10, 2022 at 11:05 AM Ben Cooksley wrote: > > > > > > > > On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol wrote: > >> > >> [Snip] > >> > >> We still haven't discussed here is how to prevent this problem from > >> happening again. > >> > >> If we don't have information about what is happening, we cannot fix > >> problems. > > > > > > Part of the issue here is that the problem only came to Sysadmin attention > > very recently, when the system ran out of disk space as a result of growing > > log files. > > It was at that point we realised we had a serious problem. > > > > Prior to that the system load hadn't climbed to dangerous levels (> number > > of CPU cores) and Apache was keeping up with the traffic, so none of our > > other monitoring was tripped. > > > > If you have any thoughts on what sort of information you are thinking of > > that would be helpful. > > We could have plots of the amount of queries we get with a KNewStuff/* > user-agent over time and their distribution. > > > It would definitely be helpful though to know when new software is going to > > be released that will be interacting with the servers as we will then be > > able to monitor for abnormalities. > > We make big announcements of every Plasma release... (?) > > >> Is there anything that could be done in this front? The issue here > >> could have been addressed months ago, we just never knew it was > >> happening. > > > > > > One possibility that did occur to me today would be for us to integrate > > some kind of killswitch that our applications would check on first > > initialisation of functionality that talks to KDE.org servers. > > This would allow us to disable the functionality in question on user > > systems. > > > > The check would only be done on first initialization to keep load low, > > while still ensuring all users eventually are affected by the killswitch > > (as they will eventually need to logout/reboot for some reason or another). > > > > The killswitch would probably work best if it had some kind of version > > check in it so we could specify which versions are disabled. > > That would allow for subsequent updates - once delivered by distributions - > > to restore the functionality (while leaving it disabled for those who > > haven't updated). > > The file we are serving here effectively is the kill switch to all of > KNewStuff. I'm a bit late to the party but for future reference I think this was/is an architectural scaling problem on the server side as much as a bug on the client. If just https load is the problem then the "hotfix" is to use a HTTP load balancer until fixes make it into the clients, killing the clients is like the last resort ever. I'm sure we have the money to afford a bunch of cloud nodes serving as selective proxy caches for a month to balance out the KNS load on the canonical server. HS
Re: Critical Denial of Service bugs in Discover
On Fri, Feb 11, 2022 at 10:22 AM Fabian Vogt wrote: > Moin, > > Am Sonntag, 6. Februar 2022, 21:54:13 CET schrieb Fabian Vogt: > > Am Sonntag, 6. Februar 2022, 19:27:11 CET schrieb Ben Cooksley: > > > On Sun, Feb 6, 2022 at 1:07 PM Fabian Vogt > wrote: > > > > The first URL is used by kfontinst.knsrc from plasma-workspace: > > > > ProvidersUrl= > https://distribute.kde.org/khotnewstuff/fonts-providers.xml > > > > > > > > The second URL is used by multiple knsrc files in my VM: > > > > aurorae.knsrc:ProvidersUrl= > https://download.kde.org/ocs/providers.xml > > > > comic.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > > > kwineffect.knsrc:ProvidersUrl= > https://download.kde.org/ocs/providers.xml > > > > kwinscripts.knsrc:ProvidersUrl= > https://download.kde.org/ocs/providers.xml > > > > kwinswitcher.knsrc:ProvidersUrl= > https://download.kde.org/ocs/providers.xml > > > > wallpaperplugin.knsrc:ProvidersUrl= > > > > https://download.kde.org/ocs/providers.xml > > > > > > This makes me incredibly sad. We had a push to eliminate all usage of > the > > > legacy download.kde.org endpoint many years ago... > > > I have now resolved the majority of these - if distributions could > please > > > pick up those patches that would be appreciated. > > > > > > Please note that I have now terminated the support on the server that > was > > > making these legacy endpoints work, so those patches are necessary to > > > restore functionality. > > ... > > It's also possible that the requests aren't actually caused by Discover > at all, > > but just something which imitates it in a DDoS attack. In that case we > couldn't > > do anything on the client-side anyway. I don't think this is very > likely, but > > until the issue was reproduced with disover it's a possibility. > > I think I have a plausible explanation for what could've caused this. > While testing a MR for the notifier, I noticed odd behaviour: It always ran > plasma-discover-update twice! > https://invent.kde.org/plasma/discover/-/merge_requests/254#note_394584 > > The reason for that is that after the update process finishes, the notifier > realizes that it's idle again and if updates are available, it will > immediately > trigger another update after the 15min idle time. Now here's the catch: If > the > system has already been idle for >=15min (which is very likely at that > point), > the idle timeout will immediately fire! This process repeats unlimited and > without delay, until the system is no longer idle or there aren't updates > available anymore. Here I have plasma-discover-update running approx. every > second, which amounts to ~4 req/s to download.kde.org. > > This is mostly mitigated by the introduction of the 3h delay between > updates > by d607e0c6f9, but not entirely. The check is only effective after the > second > iteration, which is what I observed in my testing. (One of the commits in > my MR > should address that as well.) > > One of the conditions for running into this bug is that after the automatic > updater ran, there still have to be updates available to trigger the next > run. > Initially I thought that this can mostly happen if updates fail to > download or > install, this is unfortunately not true. The notifier by default counts all > available updates, but the updater only installs offline updates. So if > there > is even a single non-offline update available, the loop continues. > Continues infinitely I assume? > > So this probably affected a lot of users who enabled automatic > installation of > updates :-/ > Do we know if any distributions flipped that switch? > > Cheers, > Fabian > > Regards, Ben
Re: Critical Denial of Service bugs in Discover
Moin, Am Sonntag, 6. Februar 2022, 21:54:13 CET schrieb Fabian Vogt: > Am Sonntag, 6. Februar 2022, 19:27:11 CET schrieb Ben Cooksley: > > On Sun, Feb 6, 2022 at 1:07 PM Fabian Vogt wrote: > > > The first URL is used by kfontinst.knsrc from plasma-workspace: > > > ProvidersUrl=https://distribute.kde.org/khotnewstuff/fonts-providers.xml > > > > > > The second URL is used by multiple knsrc files in my VM: > > > aurorae.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > > comic.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > > kwineffect.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > > kwinscripts.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > > kwinswitcher.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > > wallpaperplugin.knsrc:ProvidersUrl= > > > https://download.kde.org/ocs/providers.xml > > > > This makes me incredibly sad. We had a push to eliminate all usage of the > > legacy download.kde.org endpoint many years ago... > > I have now resolved the majority of these - if distributions could please > > pick up those patches that would be appreciated. > > > > Please note that I have now terminated the support on the server that was > > making these legacy endpoints work, so those patches are necessary to > > restore functionality. > ... > It's also possible that the requests aren't actually caused by Discover at > all, > but just something which imitates it in a DDoS attack. In that case we > couldn't > do anything on the client-side anyway. I don't think this is very likely, but > until the issue was reproduced with disover it's a possibility. I think I have a plausible explanation for what could've caused this. While testing a MR for the notifier, I noticed odd behaviour: It always ran plasma-discover-update twice! https://invent.kde.org/plasma/discover/-/merge_requests/254#note_394584 The reason for that is that after the update process finishes, the notifier realizes that it's idle again and if updates are available, it will immediately trigger another update after the 15min idle time. Now here's the catch: If the system has already been idle for >=15min (which is very likely at that point), the idle timeout will immediately fire! This process repeats unlimited and without delay, until the system is no longer idle or there aren't updates available anymore. Here I have plasma-discover-update running approx. every second, which amounts to ~4 req/s to download.kde.org. This is mostly mitigated by the introduction of the 3h delay between updates by d607e0c6f9, but not entirely. The check is only effective after the second iteration, which is what I observed in my testing. (One of the commits in my MR should address that as well.) One of the conditions for running into this bug is that after the automatic updater ran, there still have to be updates available to trigger the next run. Initially I thought that this can mostly happen if updates fail to download or install, this is unfortunately not true. The notifier by default counts all available updates, but the updater only installs offline updates. So if there is even a single non-offline update available, the loop continues. So this probably affected a lot of users who enabled automatic installation of updates :-/ Cheers, Fabian
Re: Critical Denial of Service bugs in Discover
On Thu, Feb 10, 2022 at 11:05 AM Ben Cooksley wrote: > > > > On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol wrote: >> >> [Snip] >> >> We still haven't discussed here is how to prevent this problem from >> happening again. >> >> If we don't have information about what is happening, we cannot fix problems. > > > Part of the issue here is that the problem only came to Sysadmin attention > very recently, when the system ran out of disk space as a result of growing > log files. > It was at that point we realised we had a serious problem. > > Prior to that the system load hadn't climbed to dangerous levels (> number of > CPU cores) and Apache was keeping up with the traffic, so none of our other > monitoring was tripped. > > If you have any thoughts on what sort of information you are thinking of that > would be helpful. We could have plots of the amount of queries we get with a KNewStuff/* user-agent over time and their distribution. > It would definitely be helpful though to know when new software is going to > be released that will be interacting with the servers as we will then be able > to monitor for abnormalities. We make big announcements of every Plasma release... (?) >> Is there anything that could be done in this front? The issue here >> could have been addressed months ago, we just never knew it was >> happening. > > > One possibility that did occur to me today would be for us to integrate some > kind of killswitch that our applications would check on first initialisation > of functionality that talks to KDE.org servers. > This would allow us to disable the functionality in question on user systems. > > The check would only be done on first initialization to keep load low, while > still ensuring all users eventually are affected by the killswitch (as they > will eventually need to logout/reboot for some reason or another). > > The killswitch would probably work best if it had some kind of version check > in it so we could specify which versions are disabled. > That would allow for subsequent updates - once delivered by distributions - > to restore the functionality (while leaving it disabled for those who haven't > updated). The file we are serving here effectively is the kill switch to all of KNewStuff. Aleix
Re: Critical Denial of Service bugs in Discover
On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol wrote: > [Snip] > > We still haven't discussed here is how to prevent this problem from > happening again. > > If we don't have information about what is happening, we cannot fix > problems. > Part of the issue here is that the problem only came to Sysadmin attention very recently, when the system ran out of disk space as a result of growing log files. It was at that point we realised we had a serious problem. Prior to that the system load hadn't climbed to dangerous levels (> number of CPU cores) and Apache was keeping up with the traffic, so none of our other monitoring was tripped. If you have any thoughts on what sort of information you are thinking of that would be helpful. It would definitely be helpful though to know when new software is going to be released that will be interacting with the servers as we will then be able to monitor for abnormalities. (This would have allowed us to advise on the User-Agent stuff prior to September, as well as point out potential issues with caching > Is there anything that could be done in this front? The issue here > could have been addressed months ago, we just never knew it was > happening. One possibility that did occur to me today would be for us to integrate some kind of killswitch that our applications would check on first initialisation of functionality that talks to KDE.org servers. This would allow us to disable the functionality in question on user systems. The check would only be done on first initialization to keep load low, while still ensuring all users eventually are affected by the killswitch (as they will eventually need to logout/reboot for some reason or another). The killswitch would probably work best if it had some kind of version check in it so we could specify which versions are disabled. That would allow for subsequent updates - once delivered by distributions - to restore the functionality (while leaving it disabled for those who haven't updated). > > Aleix > Thanks, Ben
Re: Critical Denial of Service bugs in Discover
On Tue, Feb 8, 2022 at 7:00 PM Ben Cooksley wrote: > > On Tue, Feb 8, 2022 at 4:24 AM Aleix Pol wrote: >> >> On Sat, Feb 5, 2022 at 10:16 PM Ben Cooksley wrote: >> > >> > Hi all, >> > >> > Over the past week or so Sysadmin has been dealing with an extremely high >> > volume of traffic directed towards both download.kde.org and >> > distribute.kde.org. >> > >> > This traffic volume is curious in so far that it is directed at two paths >> > specifically: >> > - distribute.kde.org/khotnewstuff/fonts-providers.xml >> > - download.kde.org/ocs/providers.xml >> > >> > The first path is an "internal only" host which we were redirecting a >> > legacy path to prior to the resource being relocated to cdn.kde.org. The >> > second path has been legacy for numerous years now (more than 5) and is >> > replaced by autoconfig.kde.org. >> > It is of extreme concern that these paths are still in use - especially >> > the ocs/providers.xml one. >> > >> > The volume of traffic has reached an extent that to prevent the server >> > disk filling up we have had to disable logging for those two sites. Whilst >> > dependent on the time of day the server is currently dealing with the >> > current volume of requests, which is far outside normal specifications: >> > >> > 555 requests/sec - 4.5 MB/second - 8.3 kB/request - .739199 >> > ms/request >> > >> > Analysis of a fragment of logs (comprising just a few minutes of traffic) >> > reveals the following: >> > >> > 63 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" >> > "KNewStuff/5.89.0-discoverupdate/5.23.5" >> > 64 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" >> > "KNewStuff/5.89.0-discoverupdate/5.23.4" >> > 104 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" >> > "KNewStuff/5.90.0-discoverupdate/5.23.90" >> > 105 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" >> > "KNewStuff/5.88.0-discoverupdate/5.23.5" >> >1169 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" >> > "KNewStuff/5.86.0-plasma-discover-update/" >> >1256 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" >> > "KNewStuff/5.90.0-discoverupdate/5.23.5" >> >2905 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" "Mozilla/5.0" >> > >> > 86 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 200 6773 "-" >> > "Mozilla/5.0" >> > 130 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" >> > "KNewStuff/5.89.0-discoverupdate/5.23.5" >> > 136 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" >> > "KNewStuff/5.89.0-discoverupdate/5.23.4" >> > 197 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" >> > "KNewStuff/5.88.0-discoverupdate/5.23.5" >> > 199 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" >> > "KNewStuff/5.90.0-discoverupdate/5.23.90" >> >2624 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" >> > "KNewStuff/5.86.0-plasma-discover-update/" >> >2642 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" >> > "KNewStuff/5.90.0-discoverupdate/5.23.5" >> >6117 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" >> > "Mozilla/5.0" >> > >> > This indicates that the bug lies solely within Plasma's Discover component >> > - more precisely it's updater. >> > >> > Examining the origin of these requests has indicated that some clients are >> > making requests to these paths well in excess of several times a minute >> > with a number of IP addresses appearing more 60 times in a 1 minute sized >> > sample window. >> > >> > Given that Sysadmin has raised issues with this component and it's >> > behaviour in the past, it appears that issues regarding the behaviour of >> > the OCS componentry within Discover remain unresolved. >> > >> > Due to the level of distress this is causing our systems, I am therefore >> > left with no other option other than to direct the Plasma Discover >> > developers to create and release without delay patches for all versions in >> > support, as well as for all those currently present in any actively >> > maintained distributions, that disable all OCS functionality in the >> > Discover updater. Distributions are requested to treat these patches as >> > security patches and to distribute them to users without delay. >> > >> > In 24 hours time Sysadmin will be making a posting to kde-announce >> > requesting that users immediately cease use of the Discover update client >> > as it is creating a Denial of Service attack on our infrastructure. >> >> I feel like your response here is out of proportion. >> >> Last time we had this conversation, my impression was that the problem >> was addressed for the most part. If you wanted people working on >> KNewStuff, Attica or OCS to take any actions, we needed to at the very >> least have information about you are complaining before you burst out >> into mailing lists and the likes. > > > Based on the information I had to hand back in September it was solved yes. > Our server monitoring system indicated th
Re: Critical Denial of Service bugs in Discover
On Tue, Feb 8, 2022 at 4:24 AM Aleix Pol wrote: > On Sat, Feb 5, 2022 at 10:16 PM Ben Cooksley wrote: > > > > Hi all, > > > > Over the past week or so Sysadmin has been dealing with an extremely > high volume of traffic directed towards both download.kde.org and > distribute.kde.org. > > > > This traffic volume is curious in so far that it is directed at two > paths specifically: > > - distribute.kde.org/khotnewstuff/fonts-providers.xml > > - download.kde.org/ocs/providers.xml > > > > The first path is an "internal only" host which we were redirecting a > legacy path to prior to the resource being relocated to cdn.kde.org. The > second path has been legacy for numerous years now (more than 5) and is > replaced by autoconfig.kde.org. > > It is of extreme concern that these paths are still in use - especially > the ocs/providers.xml one. > > > > The volume of traffic has reached an extent that to prevent the server > disk filling up we have had to disable logging for those two sites. Whilst > dependent on the time of day the server is currently dealing with the > current volume of requests, which is far outside normal specifications: > > > > 555 requests/sec - 4.5 MB/second - 8.3 kB/request - .739199 > ms/request > > > > Analysis of a fragment of logs (comprising just a few minutes of > traffic) reveals the following: > > > > 63 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.5" > > 64 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.4" > > 104 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.90" > > 105 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.88.0-discoverupdate/5.23.5" > >1169 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.86.0-plasma-discover-update/" > >1256 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.5" > >2905 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" "Mozilla/5.0" > > > > 86 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 200 6773 "-" > "Mozilla/5.0" > > 130 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.5" > > 136 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.4" > > 197 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.88.0-discoverupdate/5.23.5" > > 199 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.90" > >2624 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.86.0-plasma-discover-update/" > >2642 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.5" > >6117 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "Mozilla/5.0" > > > > This indicates that the bug lies solely within Plasma's Discover > component - more precisely it's updater. > > > > Examining the origin of these requests has indicated that some clients > are making requests to these paths well in excess of several times a minute > with a number of IP addresses appearing more 60 times in a 1 minute sized > sample window. > > > > Given that Sysadmin has raised issues with this component and it's > behaviour in the past, it appears that issues regarding the behaviour of > the OCS componentry within Discover remain unresolved. > > > > Due to the level of distress this is causing our systems, I am therefore > left with no other option other than to direct the Plasma Discover > developers to create and release without delay patches for all versions in > support, as well as for all those currently present in any actively > maintained distributions, that disable all OCS functionality in the > Discover updater. Distributions are requested to treat these patches as > security patches and to distribute them to users without delay. > > > > In 24 hours time Sysadmin will be making a posting to kde-announce > requesting that users immediately cease use of the Discover update client > as it is creating a Denial of Service attack on our infrastructure. > > I feel like your response here is out of proportion. > > Last time we had this conversation, my impression was that the problem > was addressed for the most part. If you wanted people working on > KNewStuff, Attica or OCS to take any actions, we needed to at the very > least have information about you are complaining before you burst out > into mailing lists and the likes. > Based on the information I had to hand back in September it was solved yes. Our server monitoring system indicated that this issue did not exist back in September - so this is new, although in the same Discover code. > > In terms of actual solutions this in would probably help to some > extent. We never merged it because it's not great design but good > results is more important than good
Re: Critical Denial of Service bugs in Discover
On Sat, Feb 5, 2022 at 10:16 PM Ben Cooksley wrote: > > Hi all, > > Over the past week or so Sysadmin has been dealing with an extremely high > volume of traffic directed towards both download.kde.org and > distribute.kde.org. > > This traffic volume is curious in so far that it is directed at two paths > specifically: > - distribute.kde.org/khotnewstuff/fonts-providers.xml > - download.kde.org/ocs/providers.xml > > The first path is an "internal only" host which we were redirecting a legacy > path to prior to the resource being relocated to cdn.kde.org. The second path > has been legacy for numerous years now (more than 5) and is replaced by > autoconfig.kde.org. > It is of extreme concern that these paths are still in use - especially the > ocs/providers.xml one. > > The volume of traffic has reached an extent that to prevent the server disk > filling up we have had to disable logging for those two sites. Whilst > dependent on the time of day the server is currently dealing with the current > volume of requests, which is far outside normal specifications: > > 555 requests/sec - 4.5 MB/second - 8.3 kB/request - .739199 > ms/request > > Analysis of a fragment of logs (comprising just a few minutes of traffic) > reveals the following: > > 63 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.5" > 64 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.4" > 104 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.90" > 105 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.88.0-discoverupdate/5.23.5" >1169 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.86.0-plasma-discover-update/" >1256 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.5" >2905 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" "Mozilla/5.0" > > 86 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 200 6773 "-" > "Mozilla/5.0" > 130 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.5" > 136 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.4" > 197 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.88.0-discoverupdate/5.23.5" > 199 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.90" >2624 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.86.0-plasma-discover-update/" >2642 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.5" >6117 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "Mozilla/5.0" > > This indicates that the bug lies solely within Plasma's Discover component - > more precisely it's updater. > > Examining the origin of these requests has indicated that some clients are > making requests to these paths well in excess of several times a minute with > a number of IP addresses appearing more 60 times in a 1 minute sized sample > window. > > Given that Sysadmin has raised issues with this component and it's behaviour > in the past, it appears that issues regarding the behaviour of the OCS > componentry within Discover remain unresolved. > > Due to the level of distress this is causing our systems, I am therefore left > with no other option other than to direct the Plasma Discover developers to > create and release without delay patches for all versions in support, as well > as for all those currently present in any actively maintained distributions, > that disable all OCS functionality in the Discover updater. Distributions are > requested to treat these patches as security patches and to distribute them > to users without delay. > > In 24 hours time Sysadmin will be making a posting to kde-announce requesting > that users immediately cease use of the Discover update client as it is > creating a Denial of Service attack on our infrastructure. I feel like your response here is out of proportion. Last time we had this conversation, my impression was that the problem was addressed for the most part. If you wanted people working on KNewStuff, Attica or OCS to take any actions, we needed to at the very least have information about you are complaining before you burst out into mailing lists and the likes. In terms of actual solutions this in would probably help to some extent. We never merged it because it's not great design but good results is more important than good design. At the moment they're in their way in but it will take time until it hits users. https://invent.kde.org/frameworks/knewstuff/-/merge_requests/141/ https://invent.kde.org/plasma/discover/-/merge_requests/165/ These were of course not the only mitigation solutions put into place back then. In fact many of them were geared towards giv
Re: Critical Denial of Service bugs in Discover
Moin, Am Sonntag, 6. Februar 2022, 19:27:11 CET schrieb Ben Cooksley: > On Sun, Feb 6, 2022 at 1:07 PM Fabian Vogt wrote: > > The first URL is used by kfontinst.knsrc from plasma-workspace: > > ProvidersUrl=https://distribute.kde.org/khotnewstuff/fonts-providers.xml > > > > The second URL is used by multiple knsrc files in my VM: > > aurorae.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > comic.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > kwineffect.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > kwinscripts.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > kwinswitcher.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > wallpaperplugin.knsrc:ProvidersUrl= > > https://download.kde.org/ocs/providers.xml > > This makes me incredibly sad. We had a push to eliminate all usage of the > legacy download.kde.org endpoint many years ago... > I have now resolved the majority of these - if distributions could please > pick up those patches that would be appreciated. > > Please note that I have now terminated the support on the server that was > making these legacy endpoints work, so those patches are necessary to > restore functionality. Does this decrease the server load noticably? On IRC you wrote that the primary offender is "KNewStuff/5.86.0-plasma-discover-update/", can you make the endpoints return an error for the top entries only? Then we'll get bug reports only for the likely cause(s) of the high traffic instead of everyone. What might help is to have a lightweight proxy in front of Apache to handle those paths in a way that doesn't stress the system much. That should probably be investigated independently of the client side, as this is entirely under our control and has immediate effects. It's also possible that the requests aren't actually caused by Discover at all, but just something which imitates it in a DDoS attack. In that case we couldn't do anything on the client-side anyway. I don't think this is very likely, but until the issue was reproduced with disover it's a possibility. > > > Given that Sysadmin has raised issues with this component and it's > > > behaviour in the past, it appears that issues regarding the behaviour of > > > the OCS componentry within Discover remain unresolved. > > > > > > Due to the level of distress this is causing our systems, I am therefore > > > left with no other option other than to direct the Plasma Discover > > > developers to create and release without delay patches for all versions > > in > > > support, as well as for all those currently present in any actively > > > maintained distributions, that disable all OCS functionality in the > > > Discover updater. Distributions are requested to treat these patches as > > > security patches and to distribute them to users without delay. > > > > Emergency workarounds for distributions might be to either not ship the KNS > > backend by not building kns-backend.so or deleting it afterwards, or > > disabling > > the discover notifier > > (/etc/xdg/autostart/org.kde.discover.notifier.desktop) > > completely. > > I have now committed patches to Discover going back to Plasma/5.18 which > disable the build of the KNS backend. > If distributions could please pick them up and distribute them as I > previously indicated that would be much appreciated. > > https://invent.kde.org/plasma/discover/-/commit/f66df3531670592960167f5060feeed6d6c792be IMO we need a more targeted approach there. The main offenders aren't running the latest version, so likely won't get those updates that quickly either. If we have more data and can pinpoint it a bit better that would at least help to speed patch delivery up. > Please note that I intend to investigate whether it is possible to serve > corrupted files from the server side that cause Discover to crash to help > alleviate the load being created by those clients. Sounds like a good way to DoS bugzilla instead and cause bad PR, both up- and downstream. On top of that, it's possible that a resulting crashloop causes an even higher frequency of requests. Cheers, Fabian > Current load being generated by this is: > > 789 requests/sec - 6.4 MB/second - 8.3 kB/request - 1.70113 > ms/request > 217 requests currently being processed, 183 idle workers > > > Cheers, > > Fabian > > > Thanks, > Ben
Re: Critical Denial of Service bugs in Discover
Hi, Am Samstag, 5. Februar 2022, 22:16:28 CET schrieb Ben Cooksley: > Hi all, > > Over the past week or so Sysadmin has been dealing with an extremely high > volume of traffic directed towards both download.kde.org and > distribute.kde.org. > > This traffic volume is curious in so far that it is directed at two paths > specifically: > - distribute.kde.org/khotnewstuff/fonts-providers.xml > - download.kde.org/ocs/providers.xml > > The first path is an "internal only" host which we were redirecting a > legacy path to prior to the resource being relocated to cdn.kde.org. The > second path has been legacy for numerous years now (more than 5) and is > replaced by autoconfig.kde.org. > It is of extreme concern that these paths are still in use - especially the > ocs/providers.xml one. > >... > > This indicates that the bug lies solely within Plasma's Discover component > - more precisely it's updater. > > Examining the origin of these requests has indicated that some clients are > making requests to these paths well in excess of several times a minute > with a number of IP addresses appearing more 60 times in a 1 minute sized > sample window. FWICT, this is caused by plasma-discover-update, which is triggered by the DiscoverNotifier service if automatic updates are enabled in kcm_updates, updates are available and the system idle for >=15min. // If the system is untouched for 1 hour, trigger the unattened update using namespace std::chrono_literals; KIdleTime::instance()->addIdleTimeout(int(std::chrono::milliseconds(15min).count())); (I wonder whether there's a bug about calling addIdleTimeout more than once. It will then invoke triggerUpdate multiple times after 15min of idle.) The Discover KNS backend creates instances for all available knsrc files, which on construction call KNSReviews::setProviderUrl with the URL defined in those files, triggering the requests. The first URL is used by kfontinst.knsrc from plasma-workspace: ProvidersUrl=https://distribute.kde.org/khotnewstuff/fonts-providers.xml The second URL is used by multiple knsrc files in my VM: aurorae.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml comic.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml kwineffect.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml kwinscripts.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml kwinswitcher.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml wallpaperplugin.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > Given that Sysadmin has raised issues with this component and it's > behaviour in the past, it appears that issues regarding the behaviour of > the OCS componentry within Discover remain unresolved. > > Due to the level of distress this is causing our systems, I am therefore > left with no other option other than to direct the Plasma Discover > developers to create and release without delay patches for all versions in > support, as well as for all those currently present in any actively > maintained distributions, that disable all OCS functionality in the > Discover updater. Distributions are requested to treat these patches as > security patches and to distribute them to users without delay. Emergency workarounds for distributions might be to either not ship the KNS backend by not building kns-backend.so or deleting it afterwards, or disabling the discover notifier (/etc/xdg/autostart/org.kde.discover.notifier.desktop) completely. Cheers, Fabian
Re: Critical Denial of Service bugs in Discover
On Sun, Feb 6, 2022 at 1:07 PM Fabian Vogt wrote: > Hi, > > Am Samstag, 5. Februar 2022, 22:16:28 CET schrieb Ben Cooksley: > > Hi all, > > > > Over the past week or so Sysadmin has been dealing with an extremely high > > volume of traffic directed towards both download.kde.org and > > distribute.kde.org. > > > > This traffic volume is curious in so far that it is directed at two paths > > specifically: > > - distribute.kde.org/khotnewstuff/fonts-providers.xml > > - download.kde.org/ocs/providers.xml > > > > The first path is an "internal only" host which we were redirecting a > > legacy path to prior to the resource being relocated to cdn.kde.org. The > > second path has been legacy for numerous years now (more than 5) and is > > replaced by autoconfig.kde.org. > > It is of extreme concern that these paths are still in use - especially > the > > ocs/providers.xml one. > > > >... > > > > This indicates that the bug lies solely within Plasma's Discover > component > > - more precisely it's updater. > > > > Examining the origin of these requests has indicated that some clients > are > > making requests to these paths well in excess of several times a minute > > with a number of IP addresses appearing more 60 times in a 1 minute sized > > sample window. > > FWICT, this is caused by plasma-discover-update, which is triggered by the > DiscoverNotifier service if automatic updates are enabled in kcm_updates, > updates are available and the system idle for >=15min. > > // If the system is untouched for 1 hour, trigger the unattened update > using namespace std::chrono_literals; > > KIdleTime::instance()->addIdleTimeout(int(std::chrono::milliseconds(15min).count())); > > (I wonder whether there's a bug about calling addIdleTimeout more than > once. > It will then invoke triggerUpdate multiple times after 15min of idle.) > That may explain why we are seeing so many requests from some IPs and very few from others. > > The Discover KNS backend creates instances for all available knsrc files, > which on construction call KNSReviews::setProviderUrl with the URL defined > in > those files, triggering the requests. > That does not sound scalable, and would certainly explain why not too long ago we found that the traffic received by autoconfig.kde.org had grown to such an extent we had to shift it to being handled by a CDN. At the time I chalked the problem up to increasing popularity of our software that included KNS functionality. > > The first URL is used by kfontinst.knsrc from plasma-workspace: > ProvidersUrl=https://distribute.kde.org/khotnewstuff/fonts-providers.xml > > The second URL is used by multiple knsrc files in my VM: > aurorae.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > comic.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > kwineffect.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > kwinscripts.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > kwinswitcher.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > wallpaperplugin.knsrc:ProvidersUrl= > https://download.kde.org/ocs/providers.xml This makes me incredibly sad. We had a push to eliminate all usage of the legacy download.kde.org endpoint many years ago... I have now resolved the majority of these - if distributions could please pick up those patches that would be appreciated. Please note that I have now terminated the support on the server that was making these legacy endpoints work, so those patches are necessary to restore functionality. > > > Given that Sysadmin has raised issues with this component and it's > > behaviour in the past, it appears that issues regarding the behaviour of > > the OCS componentry within Discover remain unresolved. > > > > Due to the level of distress this is causing our systems, I am therefore > > left with no other option other than to direct the Plasma Discover > > developers to create and release without delay patches for all versions > in > > support, as well as for all those currently present in any actively > > maintained distributions, that disable all OCS functionality in the > > Discover updater. Distributions are requested to treat these patches as > > security patches and to distribute them to users without delay. > > Emergency workarounds for distributions might be to either not ship the KNS > backend by not building kns-backend.so or deleting it afterwards, or > disabling > the discover notifier > (/etc/xdg/autostart/org.kde.discover.notifier.desktop) > completely. > I have now committed patches to Discover going back to Plasma/5.18 which disable the build of the KNS backend. If distributions could please pick them up and distribute them as I previously indicated that would be much appreciated. https://invent.kde.org/plasma/discover/-/commit/f66df3531670592960167f5060feeed6d6c792be Please note that I intend to investigate whether it is possible to serve corrupted files from the server side that cause Discover to crash
