Wow, you did a great job there Floh I guess !
I definitively agree with your approach/ideas, having the granularity at
the rigid asset bundle level or at the single file level would not be
good enough on huge game use cases.
About the "block-filesystem" I was thinking about something inspired by
modern filesystems: for example 128KB compressed blocks like in ZFS.
Along with some background asset prefetching system, that would try to
dynamically guess what are the best assets to fetch (during gameplay),
based on: what are the next/previous levels/areas, what the last recent
autotsaves are pointing to, and on automated analysis of asset dependencies.
I did not thought about the content update issue, this is certainly
something to take into consideration too.
The only catch with all of this is that legacy games (= not developed
for the Web in mind) are designed around the fact filesystem IO
operations are synchronous. It's been like this since 25+ years now...
Using a HTTP backend would make it asynchronous. There is big design
changes required ahead :/
@Sylvain, the XHR-backed FS is really interesting, but as you mention,
it needs to be run in a web worker. So this is moving the "sync/async"
issue elsewhere (not between game client and server, but between game
main thread and worker)
Le 11/03/2019 à 01:14, Floh a écrit :
What I've done in an F2P game (not a web game, but native PC) is to
not use asset bundles, but load each asset file (which are usually
somewhere between a few kbytes to dozens of kbytes big) through an
HTTP request, with a hierarchical content-hash-based local caching.
Basically a read-only content-addressable filesystem working through
standard HTTP servers.
We assumed the average player has a 2 MBit/sec connection, and wanted
no more than 20 second waiting time between map changes to download
critical data needed to start the map, and then continue to stream
data as needed while a map was running (this may have the effect that
3D models "plop in" though, but its the same idea as loading a web page).
This works quite well, but could be improved. For instance very small
files had a lot of overhead for the HTTP request/response headers, and
on the local machine, the cache was made of thousands of small files,
which is quite slow on Windows filesystems.
One thing I had in mind to improve this is to turn this into a
"block-filesystem". Instead of downloading individual files, merge all
files into a single big blob, made of many small (16..64 KBytes?)
blocks, and do the whole content-hash-addressing on blocks instead of
files. So for a new game session you would start with a
"filesystem-file" which is made of empty, invalid blocks, and while
the game is running, those blocks get downloaded and "validated" as
needed.
One problem with this approach is content updates (when new versions
of the game are released, which happens every 2..3 weeks or so). Some
of the data gets updated, files are added, or existing files can
change, not only the content but also the size. Handling this naively
would introduce problems like fragmentation in the cache file... but
one could separate "cold data" which hardly changes from "hot data"
which very likely changes between updates, and handle those
differently (or a new file starts in the "hot data" set, but slowly
moves to the "cold data" set)...
All in all I think this approach is better than both granular
"single-file downloads" on one extreme, and rigid asset-bundles on the
other extreme.
It's basically like one big asset bundle that's downloaded / updated
in small pieces as needed.
Cheers,
-Floh.
On Sunday, 10 March 2019 23:13:04 UTC+1, Gabriel CV wrote:
A technical point of view:
Games are based since years on the concept of big "Asset Bundles",
be it .pak/.zip/.bundle/whatever files. So a first option is to
try to bundle things in a smarter way, that could help improve
things a bit.
But in the end, this concept of "Asset Bundle" is probably not the
way to go for the web.
The issue with this is that such Bundles are static, and lack
granularity required by the Web limitations. They are just big
blobs of data, generally assembled manually by packager/developers
based on empirical observations and other convenience factors.
Most of the time you'll have some "common" asset bundles, "map
specific" asset bundles, etc. And quickly, due to the way games
are being developed, there is corner cases: a texture is needed,
but it come from a bundle for which 99% of the content is
unrelated to the current map/level/whatever. Sometime assets are
duplicated because of this.
While on Desktop/Mobile this OK, on the Web this is a killer issue
due to limited bandwitdth and lack of real Filesystem storage.
Now, you REALLY want to have only what is necessary at any given
time, regardless of the empirical packaging work.
I can see two way to achieve this:
- Either you have a "Dynamic Asset Bundler" system implemented on
the server. Basically, there is server code that is able to build
dependency graphs of all assets. Bundling is no more a
manual/empirical thing, but follows a precise and formal
algorithm. So, when the game asks the server that it want some
asset, the server is automatically replying with the full list of
dependent assets needed. The game then send a list of assets it
already have locally, and finally, using both those lists (what is
needed, and what is already there), the server is building a
specific bundle for that specific request. Bundles are created
dynamically and client-dependent, and no more static and
developer/packager dependent. This is a very different philosophy.
- Either you have a "Individual Asset Cache" system implemented
locally. In this case, the game don't cares at all about bundling.
When the game want an asset, he look for it in a local cache that
is quite limited in size (either in memory fs, or in Indexed db, I
don't know). If if it is not in the cache, the game fetch it from
the server and put it into the cache. If the cache is full, the
oldest stuff is being removed. And so on, for each individual
assets and dependencies that might be needed. The major issue with
this is that such server queries are asynchronous on the web, and
existing games are used to have synchronous IO operations since
the dawn of video games. So there is some major design changes
needed to achieve this.
So in both cases... there is non trivial work needed to allow
multi-GB AAA games on the web. That's not impossible though I
think, it just require some serious design.... and maybe axing a
bit the texture sizes ;)
Le dimanche 10 mars 2019 07:51:30 UTC+1,
[email protected] a écrit :
Just curious if it's currently too early to deploy
console-quality PC games to the web using wasm? Seems like
multithreading and wasm-64 will bring about the biggest
improvements in this regard.
Le dimanche 10 mars 2019 13:23:02 UTC+1, Floh a écrit :
PS: one big problem with streaming assets during gameplay are
wonky mobile connections of course. The whole idea only works
if the internet connection doesn't drop every few minutes.
On Sunday, 10 March 2019 13:17:00 UTC+1, Floh wrote:
IMHO the most important problem (before all the others) is
how do you get all that data into your game. Current
PC/console games are usually downloaded and installed
upfront before you can start playing. This is a problem
for web platforms, you don't want to let the player wait
for half an hour or longer before he can start playing,
and the browser doesn't allow to store many gigabytes as
persistent storage anyway.
Unfortunately most games engines are built around the idea
that asset data is downloaded is big blobs, and stored in
a persistent location (of the big ones, Unity may be the
best fit because it has a long history on the web).
I've been implementing on-demand-streaming of asset data
in the past a couple of times.
It works quite well *if the whole ending *and* all games
built with this engine are designed around the idea of
on-demand-data-streaming. The core problem basically moves
from "how much data do we need for the next level/region"
to "how much *new* data can we stream per second" (and
thus present to the user). Download bandwidth dictates
everything you can do in the game, but if your game is
built around the idea that you can only present as much
new (uncached) data to the user as bandwidth allows, the
overall asset size of the game can be basically infinite.
Another advantage is that player don't need to download
data they will never see. Most players only ever see 10%
of a game until they move on to another game. Why download
100% of the data upfront if only 10% are needed?
TL;DR: design your game engine's asset loading strategy
the same way Google Earth does it.
The other problems are less of technical nature I think,
but marketing and monetization. How do you sell an AAA
game on the web? How do you advertize it? Do you need
copy-protection? If yes, how would this be implemented?
That's why I think the open web also needs a new type of
game, typical console-AAA games are not a good both for
technical and non-technical reasons. Some variant of the
current F2P client/server model (give away the client for
free and control the server), basically "up-ported" mobile
games, not "down-ported AAA games".
Cheers,
-Floh.
On Sunday, 10 March 2019 07:51:30 UTC+1,
[email protected] wrote:
Just curious if it's currently too early to deploy
console-quality PC games to the web using wasm? Seems
like multithreading and wasm-64 will bring about the
biggest improvements in this regard.
--
You received this message because you are subscribed to the Google
Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
