ptrendx opened a new pull request #20212:
URL: https://github.com/apache/incubator-mxnet/pull/20212
## Description ##
This commit fixes 2 problems:
- previously in the calculation of the workspace size of NMS operator the
space for the buffer was not included, resulting in potentially too small
allocation
- because the type requested for `uint8_t`, the maximum workspace size
without setting large tensor support was 2GB, which was too small for some
cases, without any error reporting (going over that limit would make the
index_t in Shape1 overflow, resulting in negative value, which would later be
cast to size_t in the allocator and MXNet would try to allocate a huge amount
of memory, resulting in OoM). This PR changes the type of the allocation to
double so that the size limit for workspace rises to 16 GB when large tensor
support is not enabled.
## Checklist ##
### Essentials ###
- [x] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL],
[FEATURE], [DOC], etc)
- [x] Changes are complete (i.e. I finished coding on this PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
