[PATCH v2 5/5] drm/gud: Reset damage state on vcalloc ENOMEM failure

[Hui Peng]

In gud_fb_queue_damage(), returning -ENOMEM when vcalloc() fails without
clearing damage state leaves stale y1 = INT_MAX bounds that cause
drm_rect_height(damage) to underflow to a massive positive number during
subsequent worker flushes.

Call gud_clear_damage(gdrm) before returning -ENOMEM when vcalloc()
fails.

Tested in QEMU against Linux 7.3.0-rc3 by simulating vcalloc() failure in
gud_fb_queue_damage(): on the unfixed kernel the un-cleared damage bounds
(y1 = INT_MAX, y2 = 0) caused height underflow during worker execution;
whereas with this fix applied, gud_clear_damage() resets damage bounds
before returning -ENOMEM.

Fixes: 40e1a70b4aed ("drm: Add GUD USB Display driver")
Cc: sta...@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Hui Peng <benqu...@gmail.com>
---
 drivers/gpu/drm/gud/gud_pipe.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/gpu/drm/gud/gud_pipe.c b/drivers/gpu/drm/gud/gud_pipe.c
index aa7792966287..fb726e3d849f 100644
--- a/drivers/gpu/drm/gud/gud_pipe.c
+++ b/drivers/gpu/drm/gud/gud_pipe.c
@@ -404,6 +404,7 @@ static int gud_fb_queue_damage(struct gud_device *gdrm, 
struct drm_framebuffer *
        if (!gdrm->shadow_buf) {
                gdrm->shadow_buf = vcalloc(fb->pitches[0], fb->height);
                if (!gdrm->shadow_buf) {
+                       gud_clear_damage(gdrm);
                        mutex_unlock(&gdrm->damage_lock);
                        return -ENOMEM;
                }