leezu commented on a change in pull request #12042: Make check_isfinite,
check_scale optional in clip_global_norm
URL: https://github.com/apache/incubator-mxnet/pull/12042#discussion_r212812586
##########
File path: python/mxnet/gluon/utils.py
##########
@@ -115,8 +115,21 @@ def split_and_load(data, ctx_list, batch_axis=0,
even_split=True):
return [i.as_in_context(ctx) for i, ctx in zip(slices, ctx_list)]
-def clip_global_norm(arrays, max_norm):
+def clip_global_norm(arrays, max_norm, check_isfinite=True, check_scale=True):
"""Rescales NDArrays so that the sum of their 2-norm is smaller than
`max_norm`.
+
+ Parameters
+ ----------
+ arrays : list of NDArray
+ max_norm : float
+ check_isfinite : bool, default True
+ If True, check that the total_norm is finite (not nan or inf). This
+ requires a blocking .asscalar() call.
+ check_scale : bool, default True
+ If True, skip array rescaling if max_norm / total_norm >= 1. This
+ requires a blocking call. If False, rescale arrays with min(1,
Review comment:
For now the PR preserves the old default behavior. I'm happy to remove the
`check_scale` argument and to always rescale, trading off computation against
avoiding blocking calls. That would assume that it is always cheaper/better do
the rescaling than to wait for a blocking asscalar() and potentially avoid
rescaling.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
