Re: [PATCH v2] fbmem: don't allow too huge resolutions

2021-09-01 Thread Geert Uytterhoeven
On Wed, Sep 1, 2021 at 3:15 AM Tetsuo Handa wrote: > syzbot is reporting page fault at vga16fb_fillrect() [1], for > vga16fb_check_var() is failing to detect multiplication overflow. > > if (vxres * vyres > maxmem) { > vyres = maxmem / vxres; > if (vyres < yres) > return -ENOMEM;

[PATCH v2] fbmem: don't allow too huge resolutions

2021-08-31 Thread Tetsuo Handa
syzbot is reporting page fault at vga16fb_fillrect() [1], for vga16fb_check_var() is failing to detect multiplication overflow. if (vxres * vyres > maxmem) { vyres = maxmem / vxres; if (vyres < yres) return -ENOMEM; } Since no module would accept too huge resolutions where