connorgoggins commented on a change in pull request #17617: [Large Tensor]
Fixed Spatial Transformer op
URL: https://github.com/apache/incubator-mxnet/pull/17617#discussion_r381202221
##########
File path: src/operator/spatial_transformer.cc
##########
@@ -58,10 +58,10 @@ inline void BilinearSamplingForward(const Tensor<cpu, 4,
DType> &output,
const DType top_left_x_w = 1.0 - (x_real - top_left_x);
const index_t data_index = n * i_c * i_h * i_w + c * i_h * i_w +
top_left_y * i_w + top_left_x;
- index_t top_left_v = 0;
- index_t top_right_v = 0;
- index_t bottom_left_v = 0;
- index_t bottom_right_v = 0;
+ DType top_left_v = 0;
Review comment:
@ChaiBapchya changing DType to index_t for these specific variables makes
the op give incorrect on standard inputs (e.g. the inputs in the CI run) - the
values generated in the output NDArray are all integers instead of floats. This
is due to the fact that these variables do not represent the index positions
(as I also originally believed), but the underlying values at the vertices.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
