Just a little follow-up posting on the latest findings (thanks to user poltroon!) on the topic of this thread.
We recently discovered that a lot of our traffic and the -vast- majority of the web requests were caused by some LMS instances sending an insane amount of requests. Some single systems hit our servers with more than 2Mio. hits within 24h. Some "only" did like 20 requests per second for a few hours. So I decided to implement hit rate limiting on the load-balancers, rejecting the worst offenders. 'User poltroon discovered those 503 errors in his server.log file' (http://forums.slimdevices.com/showthread.php?t=105608). He was kind enough to investigate the issue on his end. What he figured out was that the hammering was caused by two instances of squeezelite running on the same machine, configured to report the same MAC address. This somehow triggered the query loop in LMS. Once he shut down one instance things went back to normal. So I went back to the log files and database. And sure enough: all of the top offenders (those sending several 100k requests per day) are using software players. Many of them no real Squeezebox at all. Now I'm not saying using SW players is evil. BUT they're much more prone to issues, as we have less control over them. And as a consequence of this higher error rate we do see above-average cost for systems which don't pay for the service... Just one number: database queries per second instantly dropped by about 40-50% after I enabled the hit rate limiting. Those probably two to three dozen installations running a broken configuration caused us to install additional database servers in the past. And that costs real money. This random sample shows another issue: 20486 (if you recognize your setup, please check your configuration: it triggered about 720k requests yesterday :-)) I doubt this user has three players in the kitchen (Kueche) and two in the living room (Wohnzimmer). These are simply multiple setups which were used over time. Software players are cheap. You install and remove them whenever you feel like it. Alas, the data stays in the database and must be processed whenever LMS requests the players list. And there are users with hundreds of players. Two of those requests hammering us are player status and pref sync queries. With an average of 50 prefs per player stored in the database you can imagine that having a request sent 20x per second for every single player can hit us hard. Ok, back to work. And please check your configurations. Thanks! +-------------------------------------------------------------------+ |Filename: Bildschirmfoto 2016-05-17 um 09.24.59.png | |Download: http://forums.slimdevices.com/attachment.php?attachmentid=20486| +-------------------------------------------------------------------+ Michael http://www.herger.net/slim-plugins - MusicArtistInfo, MusicInfoSCR, Smart Mix ------------------------------------------------------------------------ mherger's Profile: http://forums.slimdevices.com/member.php?userid=50 View this thread: http://forums.slimdevices.com/showthread.php?t=105461 _______________________________________________ squeezenetwork mailing list [email protected] http://lists.slimdevices.com/mailman/listinfo/squeezenetwork
