What I've done in an F2P game (not a web game, but native PC) is to not use asset bundles, but load each asset file (which are usually somewhere between a few kbytes to dozens of kbytes big) through an HTTP request, with a hierarchical content-hash-based local caching. Basically a read-only content-addressable filesystem working through standard HTTP servers.
We assumed the average player has a 2 MBit/sec connection, and wanted no more than 20 second waiting time between map changes to download critical data needed to start the map, and then continue to stream data as needed while a map was running (this may have the effect that 3D models "plop in" though, but its the same idea as loading a web page). This works quite well, but could be improved. For instance very small files had a lot of overhead for the HTTP request/response headers, and on the local machine, the cache was made of thousands of small files, which is quite slow on Windows filesystems. One thing I had in mind to improve this is to turn this into a "block-filesystem". Instead of downloading individual files, merge all files into a single big blob, made of many small (16..64 KBytes?) blocks, and do the whole content-hash-addressing on blocks instead of files. So for a new game session you would start with a "filesystem-file" which is made of empty, invalid blocks, and while the game is running, those blocks get downloaded and "validated" as needed. One problem with this approach is content updates (when new versions of the game are released, which happens every 2..3 weeks or so). Some of the data gets updated, files are added, or existing files can change, not only the content but also the size. Handling this naively would introduce problems like fragmentation in the cache file... but one could separate "cold data" which hardly changes from "hot data" which very likely changes between updates, and handle those differently (or a new file starts in the "hot data" set, but slowly moves to the "cold data" set)... All in all I think this approach is better than both granular "single-file downloads" on one extreme, and rigid asset-bundles on the other extreme. It's basically like one big asset bundle that's downloaded / updated in small pieces as needed. Cheers, -Floh. On Sunday, 10 March 2019 23:13:04 UTC+1, Gabriel CV wrote: > > A technical point of view: > > Games are based since years on the concept of big "Asset Bundles", be it > .pak/.zip/.bundle/whatever files. So a first option is to try to bundle > things in a smarter way, that could help improve things a bit. > > But in the end, this concept of "Asset Bundle" is probably not the way to > go for the web. > > The issue with this is that such Bundles are static, and lack granularity > required by the Web limitations. They are just big blobs of data, generally > assembled manually by packager/developers based on empirical observations > and other convenience factors. Most of the time you'll have some "common" > asset bundles, "map specific" asset bundles, etc. And quickly, due to the > way games are being developed, there is corner cases: a texture is needed, > but it come from a bundle for which 99% of the content is unrelated to the > current map/level/whatever. Sometime assets are duplicated because of this. > > While on Desktop/Mobile this OK, on the Web this is a killer issue due to > limited bandwitdth and lack of real Filesystem storage. Now, you REALLY > want to have only what is necessary at any given time, regardless of the > empirical packaging work. > > I can see two way to achieve this: > > - Either you have a "Dynamic Asset Bundler" system implemented on the > server. Basically, there is server code that is able to build dependency > graphs of all assets. Bundling is no more a manual/empirical thing, but > follows a precise and formal algorithm. So, when the game asks the server > that it want some asset, the server is automatically replying with the full > list of dependent assets needed. The game then send a list of assets it > already have locally, and finally, using both those lists (what is needed, > and what is already there), the server is building a specific bundle for > that specific request. Bundles are created dynamically and > client-dependent, and no more static and developer/packager dependent. This > is a very different philosophy. > > - Either you have a "Individual Asset Cache" system implemented locally. > In this case, the game don't cares at all about bundling. When the game > want an asset, he look for it in a local cache that is quite limited in > size (either in memory fs, or in Indexed db, I don't know). If if it is not > in the cache, the game fetch it from the server and put it into the cache. > If the cache is full, the oldest stuff is being removed. And so on, for > each individual assets and dependencies that might be needed. The major > issue with this is that such server queries are asynchronous on the web, > and existing games are used to have synchronous IO operations since the > dawn of video games. So there is some major design changes needed to > achieve this. > > So in both cases... there is non trivial work needed to allow multi-GB AAA > games on the web. That's not impossible though I think, it just require > some serious design.... and maybe axing a bit the texture sizes ;) > > Le dimanche 10 mars 2019 07:51:30 UTC+1, [email protected] a > écrit : >> >> Just curious if it's currently too early to deploy console-quality PC >> games to the web using wasm? Seems like multithreading and wasm-64 will >> bring about the biggest improvements in this regard. >> > > > Le dimanche 10 mars 2019 13:23:02 UTC+1, Floh a écrit : >> >> PS: one big problem with streaming assets during gameplay are wonky >> mobile connections of course. The whole idea only works if the internet >> connection doesn't drop every few minutes. >> >> On Sunday, 10 March 2019 13:17:00 UTC+1, Floh wrote: >>> >>> IMHO the most important problem (before all the others) is how do you >>> get all that data into your game. Current PC/console games are usually >>> downloaded and installed upfront before you can start playing. This is a >>> problem for web platforms, you don't want to let the player wait for half >>> an hour or longer before he can start playing, and the browser doesn't >>> allow to store many gigabytes as persistent storage anyway. >>> >>> Unfortunately most games engines are built around the idea that asset >>> data is downloaded is big blobs, and stored in a persistent location (of >>> the big ones, Unity may be the best fit because it has a long history on >>> the web). >>> >>> I've been implementing on-demand-streaming of asset data in the past a >>> couple of times. >>> >>> It works quite well *if the whole ending *and* all games built with this >>> engine are designed around the idea of on-demand-data-streaming. The core >>> problem basically moves from "how much data do we need for the next >>> level/region" to "how much *new* data can we stream per second" (and thus >>> present to the user). Download bandwidth dictates everything you can do in >>> the game, but if your game is built around the idea that you can only >>> present as much new (uncached) data to the user as bandwidth allows, the >>> overall asset size of the game can be basically infinite. >>> >>> Another advantage is that player don't need to download data they will >>> never see. Most players only ever see 10% of a game until they move on to >>> another game. Why download 100% of the data upfront if only 10% are needed? >>> >>> TL;DR: design your game engine's asset loading strategy the same way >>> Google Earth does it. >>> >>> The other problems are less of technical nature I think, but marketing >>> and monetization. How do you sell an AAA game on the web? How do you >>> advertize it? Do you need copy-protection? If yes, how would this be >>> implemented? >>> >>> That's why I think the open web also needs a new type of game, typical >>> console-AAA games are not a good both for technical and non-technical >>> reasons. Some variant of the current F2P client/server model (give away the >>> client for free and control the server), basically "up-ported" mobile >>> games, not "down-ported AAA games". >>> >>> Cheers, >>> -Floh. >>> >>> On Sunday, 10 March 2019 07:51:30 UTC+1, [email protected] >>> wrote: >>>> >>>> Just curious if it's currently too early to deploy console-quality PC >>>> games to the web using wasm? Seems like multithreading and wasm-64 will >>>> bring about the biggest improvements in this regard. >>>> >>> -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
