wangwei created SINGA-115:
-----------------------------
Summary: Print layer debug information in the neural net graph file
Key: SINGA-115
URL: https://issues.apache.org/jira/browse/SINGA-115
Project: Singa
Issue Type: New Feature
Reporter: wangwei
It is non-trivial to debug the code for deep learning, e.g., the BP algorithm,
the hybrid partitioning and layer implementation.
In SINGA, we print the neural net in INFO log as json string, which can be
converted into an image with the net graph (nodes are layers). This graph can
be used to check the neural net configuration, e.g., layer connection and
neural net partitioning. However, it does not collect the run time data, e.g.,
gradient norm or value norm of each layer, which is important to debug
accuracy etc. bugs.
In this ticket, we will collect the gradient and value norm of each layer and
each Param object. These information will be printed as attributes (or
sub-nodes) of the layer node in the neural net graph. Users/developers can
located the bugs by inspecting the graph after converting the json string into
an image.
Particularly, uses can set the disp_freq to 1 and running steps to a small
number, e.g., 5. Then 5 neural net graphs will be printed, one per step. The
debug option should be turned on in the job.conf file for printing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
