Division by zero may occur in rare constellation of conditions if:
1. not TrueColor mode on the client side
tight_detect_smooth_image16() and tight_detect_smooth_image32(),
defined by macro DEFINE_DETECT_FUNCTION()2, are affected.
2. if all pixels on the screen are equal, then pixels == stats[0]
Found by Linux Verification Center (linuxtesting.org) with SVACE.
Signed-off-by: Dmitry Frolov <fro...@swemel.ru>
---
ui/vnc-enc-tight.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c
index 41f559eb83..f1249ab136 100644
--- a/ui/vnc-enc-tight.c
+++ b/ui/vnc-enc-tight.c
@@ -284,6 +284,9 @@ tight_detect_smooth_image24(VncState *vs, int w, int h)
for (; c < 256; c++) { \
errors += stats[c] * (c * c); \
} \
+ if (pixels == stats[0]) { \
+ return 0; \
+ } \
errors /= (pixels - stats[0]); \
\
return errors; \
--
2.34.1
