If I understand correctly, some people get CRC errors. I have seen this many times. If someone gets the wrong file somehow, it will not re-download the correct file, but it will check the CRC, which will be wrong. The user must find the file with the CRC error and delete it.
The cause for me was... If the web server could not find the file, it gives a bogus .html file that would give a pretty web page to a user saying file not found. That .html file is named as whatever file they were trying to download. A correct web server would just give a 404 error. The game can handle that. I requested that valve change things so it will re-download a file anytime it gets a CRC error on an existing file. No response, if I recall. Also would be a problem if 2 different maps had the same name. } -----Original Message----- } From: hlds_linux-boun...@list.valvesoftware.com [mailto:hlds_linux- } boun...@list.valvesoftware.com] On Behalf Of Philipp Reddigau } Sent: Thursday, January 21, 2010 12:48 PM } To: 'Half-Life dedicated Linux server mailing list' } Subject: Re: [hlds_linux] BZip2 compression methods } } Hi, } I will try to answer all replys in one post. } } First i will describe how our FDL works. } The content server(fdl) connect all 5 minutes to all boxes an check| } someone } uploaded a file? } if there are any changes against the content server the new data will be } transfered. } So in a general time of 10 Minutes all files are up to date. } But with missing *.bz2 files. } The bz2 files are generated each night. After the first compression they } are } going into several checks to be sure that the file is ok and complete. } (Example: Maybe a bsp was not uploaded complete at fdl sync time) } On each night we have about 500 to 1000 new files including. bsp, nav, mp3 } and so on. } why so much? } It is possible that one Custom Map (same Version) differ from a other } Server } cause client upload failures or other crap. } As an example: } We have the map $2000$.bsp in 7 versions. (But it is the same Map and they } are working all) } So we have this map 7 times as bz2 because..the engine.. } Its crap if you upload any of this versions to fdl as normal bsp all } players } can play with it. Now upload any version of the *.bz2 it will not work on } all players, some players can join some not. } I hope you understand what i mean. English is not my best language ;). } } So now to your suggestions/questions: } @Bjorn } --force is needed for some integrity checks we are touch *.bz2 } Something like that. } if [ ! -e ./bz2data_diff/bsp/"$file".bz2 ]; then } touch ./bz2data_diff/bsp/"$file".bz2 } fi } We are packing only new files and delete/blacklist all bz2 of the same } file } if we reach a file version limit of 10. } } @Shane } Yes i have checked. Its compatible with bzip2. } The output is a little bit bigger than the bzip2 compressed file, because } pbzip2 breaks the source file in little pieces and each piece is } compressed } by a other core. } I think that the developers are using a really old libary to decompress. } Because of that i ask about the method that the developers are using. } concrete witch libary version are in use? } If i know that i will program a multithreaded bzip2 myself. } } Have you tried to upload a compressed pbzip2 file to a fdl? } Maybe someone can test it. } } } Mit freundlichen Grüßen, } Philipp Reddigau (GKServer) } } GKServer } Daniel Zschau } Niederfährer Straße 38 } 01662 Meißen } } } -----Ursprüngliche Nachricht----- } Von: hlds_linux-boun...@list.valvesoftware.com } [mailto:hlds_linux-boun...@list.valvesoftware.com] Im Auftrag von Shane } Turner } Gesendet: Donnerstag, 21. Januar 2010 17:38 } An: Half-Life dedicated Linux server mailing list } Betreff: Re: [hlds_linux] BZip2 compression methods } } I think we need a slightly better description of the problem you're } trying to solve. } } My interpretation of your original post is that you have a large number } of files to compress, but many cores. You tried pbzip2 (Parallel BZIP2) } and the resulting files couldn't be decompressed. } } Have you confirmed that the pbzip2 compressed files are decompressible } by normal bzip2? The page for pbzip2 indicates that they should be } compatible with bzip2 1.0.2 or newer. } } The problem could be that the game is using a bzip2 algorithm that isn't } compatible with bzip2 1.0.2. } } Shane } } On 3:59 PM, Philipp Reddigau wrote: } > Hi, } > Thank you for the reply. } > But this will compress up to 10 files at once. } > It will not compress one file on more than one core. } > It would be better and much faster if one file compressed by 8 or 16 } cores. } > } > An Example: (we are do it like your skript) } > 5000 files: } > normal method(one file at time one core): } > 7h 34m } > } > like your skript(15 files on 15 cores): !extreme load, hdd is unusable! } > 1h 37m } > } > pbzip2(8 cores): !all is right, hdd usuable! } > 1h 43m } > } > Take care at the number of cores. } > } > Mit freundlichen Grüßen, } > Philipp Reddigau (GKServer) } > } > GKServer } > Daniel Zschau } > Niederfährer Straße 38 } > 01662 Meißen } > } > } > -----Ursprüngliche Nachricht----- } > Von: hlds_linux-boun...@list.valvesoftware.com } > [mailto:hlds_linux-boun...@list.valvesoftware.com] Im Auftrag von kama } > Gesendet: Donnerstag, 21. Januar 2010 14:17 } > An: Half-Life dedicated Linux server mailing list } > Betreff: Re: [hlds_linux] BZip2 compression methods } > } > } > } > On Thu, 21 Jan 2010, Philipp Reddigau wrote: } > } > } >> Hi, } >> we are using a automatic *.bz2 for our content servers (fastdownload) } on } >> nix/lenny. } >> It works fine with bzip2 --compress --keep --force --best "$file" } >> But we have one Problem..we have many files to compress and one bzip2 } is } >> } > not } > } >> enough. } >> So we have tried pbzip2 after compress join on server "Could not CRC } >> blabla". } >> So the game cannot decompress the file.. } >> I there a way to get it clean with any other compress program? } >> (multithreaded) } >> Anyone knows witch exact format is required, that the games can } decompress } >> it? } >> I cannot find any information about the decompression. } >> } > maybe do a script that does it for you instead... } > } > This script will keep you at max 10 instances of bzip: } > } > -- snip-- } > #!/usr/local/bin/bash } > } > max } > } > for file in $* } > do } > bzip2 --compress --keep --force --best "$file"& } > while [ `ps aux | grep bzip | awk 'END { print NR }'` -ge $max ] } > do } > sleep 1 } > done } > done } > -- snap -- } > } > This is just an example. There are probably ways to tune it and make it } > better. } > } > /Bjorn } > } > _______________________________________________ } > To unsubscribe, edit your list preferences, or view the list archives, } > please visit: } > http://list.valvesoftware.com/mailman/listinfo/hlds_linux } > } > } > } > } } } _______________________________________________ } To unsubscribe, edit your list preferences, or view the list archives, } please visit: } http://list.valvesoftware.com/mailman/listinfo/hlds_linux } } } _______________________________________________ } To unsubscribe, edit your list preferences, or view the list archives, } please visit: } http://list.valvesoftware.com/mailman/listinfo/hlds_linux _______________________________________________ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux
