I like the idea of dynamically loading only what is necessary, call it smart data streaming or something similiar. Came across this today, has anybody heard of it? It's a game asset loading library that was made with large file sizes deployed to the web in mind. It's a few years old, but might be a part of the solution to this issue https://smus.com/game-asset-loader/
On Sun, Mar 10, 2019 at 6:14 PM Floh <[email protected]> wrote: > What I've done in an F2P game (not a web game, but native PC) is to not > use asset bundles, but load each asset file (which are usually somewhere > between a few kbytes to dozens of kbytes big) through an HTTP request, with > a hierarchical content-hash-based local caching. Basically a read-only > content-addressable filesystem working through standard HTTP servers. > > We assumed the average player has a 2 MBit/sec connection, and wanted no > more than 20 second waiting time between map changes to download critical > data needed to start the map, and then continue to stream data as needed > while a map was running (this may have the effect that 3D models "plop in" > though, but its the same idea as loading a web page). > > This works quite well, but could be improved. For instance very small > files had a lot of overhead for the HTTP request/response headers, and on > the local machine, the cache was made of thousands of small files, which is > quite slow on Windows filesystems. > > One thing I had in mind to improve this is to turn this into a > "block-filesystem". Instead of downloading individual files, merge all > files into a single big blob, made of many small (16..64 KBytes?) blocks, > and do the whole content-hash-addressing on blocks instead of files. So for > a new game session you would start with a "filesystem-file" which is made > of empty, invalid blocks, and while the game is running, those blocks get > downloaded and "validated" as needed. > > One problem with this approach is content updates (when new versions of > the game are released, which happens every 2..3 weeks or so). Some of the > data gets updated, files are added, or existing files can change, not only > the content but also the size. Handling this naively would introduce > problems like fragmentation in the cache file... but one could separate > "cold data" which hardly changes from "hot data" which very likely changes > between updates, and handle those differently (or a new file starts in the > "hot data" set, but slowly moves to the "cold data" set)... > > All in all I think this approach is better than both granular "single-file > downloads" on one extreme, and rigid asset-bundles on the other extreme. > > It's basically like one big asset bundle that's downloaded / updated in > small pieces as needed. > > Cheers, > -Floh. > > > On Sunday, 10 March 2019 23:13:04 UTC+1, Gabriel CV wrote: >> >> A technical point of view: >> >> Games are based since years on the concept of big "Asset Bundles", be it >> .pak/.zip/.bundle/whatever files. So a first option is to try to bundle >> things in a smarter way, that could help improve things a bit. >> >> But in the end, this concept of "Asset Bundle" is probably not the way to >> go for the web. >> >> The issue with this is that such Bundles are static, and lack granularity >> required by the Web limitations. They are just big blobs of data, generally >> assembled manually by packager/developers based on empirical observations >> and other convenience factors. Most of the time you'll have some "common" >> asset bundles, "map specific" asset bundles, etc. And quickly, due to the >> way games are being developed, there is corner cases: a texture is needed, >> but it come from a bundle for which 99% of the content is unrelated to the >> current map/level/whatever. Sometime assets are duplicated because of this. >> >> While on Desktop/Mobile this OK, on the Web this is a killer issue due to >> limited bandwitdth and lack of real Filesystem storage. Now, you REALLY >> want to have only what is necessary at any given time, regardless of the >> empirical packaging work. >> >> I can see two way to achieve this: >> >> - Either you have a "Dynamic Asset Bundler" system implemented on the >> server. Basically, there is server code that is able to build dependency >> graphs of all assets. Bundling is no more a manual/empirical thing, but >> follows a precise and formal algorithm. So, when the game asks the server >> that it want some asset, the server is automatically replying with the full >> list of dependent assets needed. The game then send a list of assets it >> already have locally, and finally, using both those lists (what is needed, >> and what is already there), the server is building a specific bundle for >> that specific request. Bundles are created dynamically and >> client-dependent, and no more static and developer/packager dependent. This >> is a very different philosophy. >> >> - Either you have a "Individual Asset Cache" system implemented locally. >> In this case, the game don't cares at all about bundling. When the game >> want an asset, he look for it in a local cache that is quite limited in >> size (either in memory fs, or in Indexed db, I don't know). If if it is not >> in the cache, the game fetch it from the server and put it into the cache. >> If the cache is full, the oldest stuff is being removed. And so on, for >> each individual assets and dependencies that might be needed. The major >> issue with this is that such server queries are asynchronous on the web, >> and existing games are used to have synchronous IO operations since the >> dawn of video games. So there is some major design changes needed to >> achieve this. >> >> So in both cases... there is non trivial work needed to allow multi-GB >> AAA games on the web. That's not impossible though I think, it just require >> some serious design.... and maybe axing a bit the texture sizes ;) >> >> Le dimanche 10 mars 2019 07:51:30 UTC+1, [email protected] a >> écrit : >>> >>> Just curious if it's currently too early to deploy console-quality PC >>> games to the web using wasm? Seems like multithreading and wasm-64 will >>> bring about the biggest improvements in this regard. >>> >> >> >> Le dimanche 10 mars 2019 13:23:02 UTC+1, Floh a écrit : >>> >>> PS: one big problem with streaming assets during gameplay are wonky >>> mobile connections of course. The whole idea only works if the internet >>> connection doesn't drop every few minutes. >>> >>> On Sunday, 10 March 2019 13:17:00 UTC+1, Floh wrote: >>>> >>>> IMHO the most important problem (before all the others) is how do you >>>> get all that data into your game. Current PC/console games are usually >>>> downloaded and installed upfront before you can start playing. This is a >>>> problem for web platforms, you don't want to let the player wait for half >>>> an hour or longer before he can start playing, and the browser doesn't >>>> allow to store many gigabytes as persistent storage anyway. >>>> >>>> Unfortunately most games engines are built around the idea that asset >>>> data is downloaded is big blobs, and stored in a persistent location (of >>>> the big ones, Unity may be the best fit because it has a long history on >>>> the web). >>>> >>>> I've been implementing on-demand-streaming of asset data in the past a >>>> couple of times. >>>> >>>> It works quite well *if the whole ending *and* all games built with >>>> this engine are designed around the idea of on-demand-data-streaming. The >>>> core problem basically moves from "how much data do we need for the next >>>> level/region" to "how much *new* data can we stream per second" (and thus >>>> present to the user). Download bandwidth dictates everything you can do in >>>> the game, but if your game is built around the idea that you can only >>>> present as much new (uncached) data to the user as bandwidth allows, the >>>> overall asset size of the game can be basically infinite. >>>> >>>> Another advantage is that player don't need to download data they will >>>> never see. Most players only ever see 10% of a game until they move on to >>>> another game. Why download 100% of the data upfront if only 10% are needed? >>>> >>>> TL;DR: design your game engine's asset loading strategy the same way >>>> Google Earth does it. >>>> >>>> The other problems are less of technical nature I think, but marketing >>>> and monetization. How do you sell an AAA game on the web? How do you >>>> advertize it? Do you need copy-protection? If yes, how would this be >>>> implemented? >>>> >>>> That's why I think the open web also needs a new type of game, typical >>>> console-AAA games are not a good both for technical and non-technical >>>> reasons. Some variant of the current F2P client/server model (give away the >>>> client for free and control the server), basically "up-ported" mobile >>>> games, not "down-ported AAA games". >>>> >>>> Cheers, >>>> -Floh. >>>> >>>> On Sunday, 10 March 2019 07:51:30 UTC+1, [email protected] >>>> wrote: >>>>> >>>>> Just curious if it's currently too early to deploy console-quality PC >>>>> games to the web using wasm? Seems like multithreading and wasm-64 will >>>>> bring about the biggest improvements in this regard. >>>>> >>>> -- > You received this message because you are subscribed to the Google Groups > "emscripten-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
