Does this mean we could stick our own (caching) HTTP proxy in front of all the servers? This would mean all our servers would be updating over the LAN, instead of having to go out to a remote content server (which may or may not be having issues at any given time).
On Fri, Dec 16, 2011 at 4:58 PM, Fletcher Dunn <[email protected]>wrote: > I'm not an expert on the new content system, but I know that alternatives > like this were considered. Widespread ISP proxy support and the ability to > temporarily expand bandwidth when Skyrim launches, etc are important > factors that make rsync (or pretty much any custom protocol) undesirable. > The new content system is based on HTTP fetching of "chunks." It also > dynamically grabs from different servers to automatically load balance. > And no matter what the delivery protocol, the tool itself has some issues, > as you've mentioned. The new delivery system has a new update tool. We're > hoping to migrate TF2 over to the new content system sometime next year. > > -----Original Message----- > From: John Schoenick [mailto:[email protected]] > Sent: Friday, December 16, 2011 1:49 PM > To: Half-Life dedicated Linux server mailing list > Cc: Fletcher Dunn > Subject: Re: [hlds_linux] (ATTN: Valve) steam.inf missing and other > "incomplete update" problems > > Are there any chances that we could get a rsync daemon on the new CDN > servers to sync public repos? It's proven to be terribly reliable, supports > renaming and file moves, and has modes to checksum all files, manage > permissions, and so on. In fact, I would go as far as saying all that > hldsupdatetool does, even if greatly improved with the new CDN, is act as a > less featureful rsync. > > Additionally it would add all sorts of flexibility to update scripts. > Telling when an update is optional vs required, moving files into place > after they're all downloaded, detecting if this update is safe to run > without shutting the server down first or locking it to one map, and so on. > It would make our lives much, much easier. > > As for improving hldsupdatetool in the meantime, the issues I've noticed > (nemrun works around some of these): > - As noted, without -verify_all, it will miss files fairly regularly > (protip: if you add -verify_all to nemrun's params it will always invoke > the updatetool as such) > - The tool will often give 'command aborted' or 'connection reset'. > Sometimes mid-update, though *often* immediately after reaching 100%, > making scripts that depend on the return value unable to tell if the update > completed > - Sometimes steam.inf just vanishes. The update tool deletes files before > downloading new ones, so maybe this is a varient of the non-verify-all > problems > - Sometimes the tool seems to fail to detect timeouts, and will sit on > 'updating files' indefinitely (hours) until you kill it. > - The tool seems to not pick the best content server except on first > startup, meaning on subsequent updates it will connect to severely > overloaded servers. Deleting the clientregistry.blob prior to each update > gives much better server selection. > > Also, if there's anything you would like nemrun and similar tools to do or > not do to make your lives easier, do let us know! > > - John > > On 12/16/2011 11:45 AM, Fletcher Dunn wrote: > > Sorry, I shouldn't have joined in with the shooting-from-the-hip > debugging. I tossed out a troubleshooting suggestion, which I probably > shouldn't have done. I was just looking for something that might be > different from this update compared to previous ones, to cause the problem. > Looks like everybody but the original poster and me already knew that > steam.inf just doesn't update properly and it's a longstanding problem. > > > > The update process...could use some improvement. We know. It will > happen when we convert the entire game to the new content delivery system. > > > > Thanks, > > - Fletch > > > > -----Original Message----- > > From: [email protected] > > [mailto:[email protected]] On Behalf Of > > PharaohsPaw > > Sent: Friday, December 16, 2011 9:05 AM > > To: HLDS Linux > > Subject: [hlds_linux] (ATTN: Valve) steam.inf missing and other > > "incomplete update" problems > > > > Although I've said much of this already I'm making a separate post in > its own topic in hopes that Valve will give it some serious consideration, > and more importantly, do something about it to fix the problem. I am > thankful that Valve staff does pay attention to this mailing list. > > > > The subject of this post is a problem that has been around for a long > time. I have been operating public gameservers off and on since 2006 and > have seen this occur with some update releases pretty much since then. I > am not the first person to say this happens, and I am also pretty sure this > is not the first time I had something to say about it here. > > > > In short, the problem is incomplete updates. It seems that the > (gamedir)/steam.inf file is the file most often left out of updates, but > sometimes other updated or new files are involved. Sometimes (apparently) > files that an "updated" server will crash without are missed. > > > > SCOPE OF PROBLEM > > I should point out that I am specifically referring only to update > scenarios where the updater for a particular gameserver "tree" initiates an > update, pulls in files, AND RECEIVES A SUCCESSFUL "HLDS Installation Is Up > to Date" MESSAGE FROM THAT MASTER AND THE STEAM BINARY RESPONSIBLE FOR > UPDATING THE SERVER FILES EXITS WITH A 0 RETURN CODE when there are > actually files missing and the server cannot run properly afterwards as a > result. > > > > The scenario in question also involves NOT using -verify_all among the > arguments passed to the steam binary to perform the update. > > > > Whether the server in question is just using srcds_run with -autoupdate > among the command line arguments, or using nemrun or any other > properly-written script isn't important. > > > > While it is possible to have incorrectly-written scripts to run and/or > update our servers, that isn't what I'm talking about here. Nemrun and > other properly-written update scripts do not run "./steam -command update > -game tf -dir ." (or whatever) only once and ASSUME the update was > successful. We can't do that because it is fairly common to get a > connection reset from whichever master we are pulling the update from > before it is complete. It has to check the return code from the steam > binary doing the update, and if it isn't 0, you have to repeat until it IS > 0. > > > > Also, it needs to be understood by anyone at Valve investigating this > problem that neither nemrun, nor any other script (including your own > srcds_run with used with -autoupdate) that I'm talking about does anything > more than call the steam binary with "-command update". Nemrun in > updatedaemon mode may have its own implementation of checking in with steam > to see if an update is needed, but it does comply with your protocols, and > even when it does start an update, it only does it because your masters > said a required update was available. And it calls the same steam binary > to do the actual updating that srcds_run does. > > > > Guess why the steam.inf file is so important? This isn't specific to > nemrun, by the way. It tells the dedicated server which version to report > to the steam masters. So its contents (or existence) means everything to > that dedicated server when it checks in with the masters. You could be > 100% in sync with the latest dedicated server files, and edit steam.inf to > show an older version, and the next time your server checks in with the > masters it is going to tell you that you need to update. And until your > steam.inf file has the same patch level/version that the masters think is > the latest required version, your server will keep telling you to restart > for the latest update. The same is true if the file isn't there at all. > > So it's not just nemrun that needs this file. It needs it for the same > reasons the gameserver itself does. > > > > The nemrun scripts are out there for Valve and anybody else to look at. > > Please look at them before you say nemrun is the problem. It isn't. > > > > So what IS going on? As far as I'm concerned we have a couple of > different issues here: > > > > 1. The steam binary can remove the existing steam.inf file while > updating a server, even if it doesn't have an updated one to replace it > with. > > > > 2. The steam binary will exit with a 0 return code and claim that the > HLDS Installation is Up To Date when the server hasn't been fully updated. > > > > Anyone seeing a pattern yet? > > > > STEAM BINARY > > > > Now, let's talk about distributing updates among your masters. Here are > a couple questions anyone thinking about this rationally should ask: > > > > 1. Why does a master server tell you that an update is available if it > doesn't have ALL of the updated files yet? If it doesn't have all the > updated files, it should not be telling you to restart your server UNTIL IT > DOES. > > > > 2. Why does the steam binary exit with a 0 RETURN CODE (successful) AND > EVEN TELL YOU 'HLDS Installation is Up To Date' if it doesn't have all the > files that got updated? > > > > > > SUGGESTIONS > > I don't have visibility of the inner workings of Valve's content > distribution system and I am not aware of whatever policies and procedures > you may have for releasing updates. (and I am not sure I would want to > either, heh). So I have to admit I am making some guesses here. But I > have a few suggestions that I think would be relevant. > > > > 1. Look at the "protocol" used by your masters, or maybe just policies > > and procedures for deploying updates onto them, which control when to > > start issuing Server Out of Date messages to dedicated servers, so > > that they will absolutely not under any circumstance start telling the > > dedicated servers heartbeating in that they need to restart/update > > UNTIL IT HAS > > *ALL* OF THE UPDATED FILES READY TO SERVE. > > > > 2. If files and directories that need to be pushed out as updated > content have to be "flagged" by the folks preparing the updates for the > masters to know which files it is supposed to push out to the dedicated > servers for an update, HAVE SOMEONE QA CHECK THE FLAGGED FILES/DIRS LIST > before the update release is approved (and the master servers start using > it) to make sure files weren't left off the list. ESPECIALLY THE STEAM.INF > FILE. The steam.inf file is ALWAYS updated for mandatory updates. So > checking the updated files list should ALWAYS require making sure steam.inf > is marked as a file to push out. > > > > 3. It is possible that nemrun or other scripts like it are talking > indirectly to a master (through an API) that isn't "in sync" with whichever > master server the steam binary will talk to to pull in the update. This is > still not nemrun's fault. The nemrun script is using SteamAPI now to see > if an update is available. > > (https://api.steampowered.com). If this API is telling "anyone that > asks" > > that a required update is out before any (AND ALL) of the master servers > the steam binary will actually pull the update in from has the complete set > of update files ready to serve, then this needs to be fixed. My suggestion > would be that api.steampowered.com be the LAST thing you guys push to - > ie, only after ALL of the steam masters that will be serving updated > content have 100% of the update and are ready to serve it. So the API host > won't say there is an update until it's ready to be served. > > > > CONCLUSION > > Rather than just telling people "-verify_all" is the only recommended > > or supported way to update your gameserver, find and fix the problems > > that make a "./steam -command update" fail sometimes without > > -verify_all added > > -- even if only to help yourselves out with the load on the masters when > updates get released. I suspect the answer lies in one or more of the > items noted above. > > > > People don't like to use -verify_all to get updates because it takes > FOREVER to finish. Doing an update without -verify_all usually only takes > a minute or so - whether doing that gets you a complete updated server or > not depends entirely on what the steam binary pulls in. Nothing else. > > You can't blame nemrun or something else when the steam binary exits > with a 0 return code and says "HLDS Installation Up To Date". > > > > Cheers. > > > > _______________________________________________ > > To unsubscribe, edit your list preferences, or view the list archives, > please visit: > > https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux > > > > _______________________________________________ > > To unsubscribe, edit your list preferences, or view the list archives, > please visit: > > https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux > > > _______________________________________________ > To unsubscribe, edit your list preferences, or view the list archives, > please visit: > https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux > _______________________________________________ To unsubscribe, edit your list preferences, or view the list archives, please visit: https://list.valvesoftware.com/cgi-bin/mailman/listinfo/hlds_linux
