gemini-code-assist[bot] commented on PR #18871: URL: https://github.com/apache/tvm/pull/18871#issuecomment-3998877262
## Summary of Changes Hello, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces significant performance optimizations for the WebGPU runtime, primarily aimed at improving the efficiency of LLM decode operations. By implementing batched GPU command submission and comprehensive caching strategies for various GPU objects like uniform buffers, bind groups, and shape tuples, the overhead associated with frequent GPU interactions and object creation is substantially reduced. Additionally, a staging buffer pooling mechanism further streamlines data transfer, and a critical padding bug in GPU memory operations has been resolved, leading to a more robust and performant WebGPU execution environment. ### Highlights * **Batched GPU Dispatch**: Compute dispatches are now batched into a single GPUCommandEncoder, which is flushed only on sync or readback operations. This significantly reduces JavaScript-to-GPU transition overhead, especially beneficial during LLM decode. * **Object Caching**: Introduced caching mechanisms for uniform buffers (FIFO/512), bind groups (FIFO/256), and shape tuples to minimize redundant GPU object creation and FFI calls. * **Staging Buffer Pooling**: A pool of MAP_READ staging buffers has been implemented to avoid the overhead of creating and destroying these buffers for each GPU-to-CPU copy operation. * **Bug Fix**: A self-assignment bug related to padding in the `deviceCopyToGPU` function has been corrected, ensuring proper data handling. * **WebGPU Feature Support**: The `timestamp-query` WebGPU feature is now conditionally requested if supported by the adapter, potentially enabling more precise performance measurements. 🧠**New Feature in Public Preview:** You can now enable **Memory** to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.** <details> <summary><b>Changelog</b></summary> * **web/src/runtime.ts** * Added a private `shapeTupleCache` map to store TVMObject shape tuples. * Updated the `dispose()` method to iterate and dispose of all cached shape tuples before clearing the cache. * Modified `makeShapeTuple()` to check the cache for existing shape tuples, returning a cached instance if found, or creating and caching a new one while detaching it from the current scope. * **web/src/webgpu.ts** * Updated `detectGPUDevice` to include `timestamp-query` in `requiredFeatures` if the adapter supports it. * Introduced new private properties for caching: `uniformBufferCache`, `readStagingBufferPool`, `pendingRead`, `pendingEncoder`, and `bindGroupCache`. * Added a `flushCommands()` method to finish and submit the `pendingEncoder` if active. * Modified `dispose()` to call `flushCommands()` and destroy/clear all newly introduced caches (uniform buffers, bind groups, read staging buffers). * Updated `sync()` to call `flushCommands()` and await any `pendingRead` promises before `onSubmittedWorkDone()`. * Modified `deviceCopyToGPU` (first instance), `deviceCopyFromGPU`, and `deviceCopyWithinGPU` to call `flushCommands()` before their respective operations to preserve execution order. * Refactored shader submission logic within `createShaderFunc` to use a single `pendingEncoder` for batching compute passes. * Implemented uniform buffer caching within shader submission, using a FIFO eviction policy for up to 512 entries and invalidating related bind group cache entries upon eviction. * Implemented bind group caching within shader submission, using a FIFO eviction policy for up to 256 entries, with keys derived from pipeline ID, buffer pointers, and uniform key. * Added a `flushCommands()` call and `bindGroupCache.clear()` in `freeBuffer()` to ensure proper cleanup and cache invalidation when a buffer is destroyed. * Fixed a padding self-assignment bug in `deviceCopyToGPU` (second instance) by correctly assigning the padded array. * Added `getReadStagingBuffer()` and `returnReadStagingBuffer()` methods to manage a pool of `MAP_READ` staging buffers, using a first-fit-by-size strategy and evicting the smallest buffer if the pool is full. * Updated `deviceCopyFromGPU` to utilize the new staging buffer pool and chain `mapAsync` promises via `pendingRead` for asynchronous readbacks. </details> <details> <summary><b>Using Gemini Code Assist</b></summary> <br> The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips. <b>Invoking Gemini</b> You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page. Feature | Command | Description --- | --- | --- Code Review | `/gemini review` | Performs a code review for the current pull request in its current state. Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state. Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments. Help | `/gemini help` | Displays a list of available commands. <b>Customization</b> To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). <b>Limitations & Feedback</b> Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG). <b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b> </details> [^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
