To add context here: The problem appears to show up only after running in production for an hour or two. During that time we will have created thousands of isolates to handle millions of requests.
But the problem seems to affect *new* isolates, even when those isolates are loaded with applications that had been loaded into previous isolates without problems. Startup of an application should be 100% deterministic since we disallow any I/O during startup, but we're seeing that after the host has been running a while, new isolates are showing much higher "external memory" on startup. (E.g. 400MB external memory, but we enforce a 128MB limit on the whole isolate.) We observed that the wasm native module cache causes identical wasm modules to be shared across isolates, and that wasm lazy compilation causes memory usage of a wasm module -- as accounted by all isolates that have loaded it -- to change. Could it be that there is a memory leak in lazy compilation, such that these shared cached modules are gradually growing over time, to the point where new isolates that try to load these modules are being hit with extremely high "external memory" numbers right off the bat? -Kenton On Mon, Jan 13, 2025 at 5:31 PM Erik Corry <erikco...@chromium.org> wrote: > It looks like it's related to shared objects between isolates. Is there a > newer document than > https://docs.google.com/document/d/18lYuaEsDSudzl2TDu-nc-0sVXW7WTGAs14k64GEhnFg/edit?usp=drivesdk > that describes how this works today? In particular cross-isolate GCs? > > On Mon, 13 Jan 2025, 15:25 Jakob Kummerow, <jkumme...@chromium.org> wrote: > >> Sounds like a bug, but without more details (or a repro) I don't have a >> more specific guess than that. >> >> If you're desperate, you could try to bisect it (even with a flaky >> repro). Or review the ~500 changes between those branches: >> https://chromium.googlesource.com/v8/v8/+log/branch-heads/13.1..branch-heads/13.2?n=10000 >> >> >> On Mon, Jan 13, 2025 at 2:48 PM 'Dan Lapid' via v8-dev < >> v8-dev@googlegroups.com> wrote: >> >>> Hi, >>> In V8 13.2 and 13.3 we see wasm isolates external memory usage blowing >>> up sometimes (up to gigabytes). >>> Under V8 13.1 the same code would never ever use more than 80-100MB >>> The issue doesn't happen every time for the same wasm bytecode. It >>> doesn't even reproduce locally. >>> But some significant percentage of the time it does happen. >>> This has only started happening in 13.2, what are we missing? Should we >>> be enabling/disabling some flags? >>> It also seems that 13.3 is significantly worse in terms of error rate. >>> The problem happens under "--liftoff-only". >>> We use pointer compression but not sandbox. >>> We've tried enabling --turboshaft-wasm in 13.1 and the problem did not >>> reproduce. >>> Has anything changed that we need to adapt to? >>> Would really appreciate your help! >>> >>> -- >>> >> -- -- v8-dev mailing list v8-dev@googlegroups.com http://groups.google.com/group/v8-dev --- You received this message because you are subscribed to the Google Groups "v8-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/v8-dev/CAJouXQn%3DkGONtBJkkdJa4TGXobsaJ5noaWOrQ05Q%3DuTQVWssiA%40mail.gmail.com.
