[SYSTEMML-618] Stabilizing the `two_layer_affine_l2_net` grad check. The example `two_layer_affine_l2_net` network uses a ReLU nonlinearity, could result in a false-negative in which the test fails due to a kink being crossed in the ReLU nonlinearity. This occurs when the tests, f(x-h) and f(x+h), end up on opposite sides of the zero threshold of max(0, fx). This change aims to stabilize the test, while still allowing for the usage of the ReLU nonlinearity. Although we have stabilized it, we will still remove it from automated testing to avoid false failures. In the future, we can explicitly check for this scenario and rerun the test automatically. For manual testing, simply rerun the tests.
Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/commit/42ebdef5 Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/tree/42ebdef5 Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/diff/42ebdef5 Branch: refs/heads/master Commit: 42ebdef5d38ad421e2f478eefb8bea535d402309 Parents: fb53098 Author: Mike Dusenberry <[email protected]> Authored: Fri Apr 28 17:51:13 2017 -0700 Committer: Mike Dusenberry <[email protected]> Committed: Fri Apr 28 17:54:14 2017 -0700 ---------------------------------------------------------------------- scripts/nn/test/grad_check.dml | 5 +++-- scripts/nn/test/run_tests.dml | 10 +++++++++- 2 files changed, 12 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/42ebdef5/scripts/nn/test/grad_check.dml ---------------------------------------------------------------------- diff --git a/scripts/nn/test/grad_check.dml b/scripts/nn/test/grad_check.dml index f3bc9a7..e767eee 100644 --- a/scripts/nn/test/grad_check.dml +++ b/scripts/nn/test/grad_check.dml @@ -1612,11 +1612,12 @@ two_layer_affine_l2_net = function() { M = 10 # number of hidden neurons [W1, b1] = affine::init(D, M) [W2, b2] = affine::init(M, yD) + W2 = W2 / sqrt(2) # different initialization, since being fed into l2 loss, instead of relu # Optimize for short "burn-in" time to move to characteristic # mode of operation and unmask any real issues. print(" - Burn-in:") - lr = 0.0001 + lr = 0.01 decay = 0.99 for(i in 1:5) { # Compute forward and backward passes of net @@ -1635,7 +1636,7 @@ two_layer_affine_l2_net = function() { [pred, loss, dX, dW1, db1, dW2, db2] = two_layer_affine_l2_net_run(X, y, W1, b1, W2, b2) # Grad check - h = 1e-5 + h = 1e-6 print(" - Grad checking X.") for (i in 1:2) { for (j in 1:ncol(X)) { http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/42ebdef5/scripts/nn/test/run_tests.dml ---------------------------------------------------------------------- diff --git a/scripts/nn/test/run_tests.dml b/scripts/nn/test/run_tests.dml index d8173a9..b061d26 100644 --- a/scripts/nn/test/run_tests.dml +++ b/scripts/nn/test/run_tests.dml @@ -60,7 +60,15 @@ grad_check::tanh() print("") # Example model -grad_check::two_layer_affine_l2_net() +# NOTE: This could result in a false-negative in which the test fails +# due to a kink being crossed in the ReLU nonlinearity. This occurs +# when the tests, f(x-h) and f(x+h), end up on opposite sides of the +# zero threshold of max(0, fx). Although we have stabilized it, we +# will still remove it here to avoid false failures during automated +# testing. In the future, we can explicitly check for this scenario +# and rerun the test automatically. For manual testing, simply +# rerun the tests. +#grad_check::two_layer_affine_l2_net() print("") print("---")
