After testing with upstream kernel, the problem seems to go away with kernel 4.8.7.
The commits related to this bug are probably the following:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=a8006bd915095c40098bfcb684cc2bdfb414dc0d
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/drivers/usb/host?h=v4.8.7&id=b9aa0a7290f53bf2158e3e5b9a306f1006f3cc91
Will wait and report when the next kernel is uploaded to debian unstable.
