Added: websites/staging/singa/trunk/content/v0.2.0/test.html ============================================================================== --- websites/staging/singa/trunk/content/v0.2.0/test.html (added) +++ websites/staging/singa/trunk/content/v0.2.0/test.html Tue Apr 12 06:24:50 2016 @@ -0,0 +1,436 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-12 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160412" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – Performance Test and Feature Extraction</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../docs/index.html" title="English">English</a> +</li> + <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">Performance Test and Feature Extraction</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>Performance Test and Feature Extraction</h1> +<hr /> +<p>Once SINGA finishes the training of a model, it would checkpoint the model parameters into disk files under the <a href="checkpoint.html">checkpoint folder</a>. Model parameters can also be dumped into this folder periodically during training if the [checkpoint configuration[(checkpoint.html) fields are set. With the checkpoint files, we can load the model parameters to conduct performance test, feature extraction and prediction against new data.</p> +<p>To load the model parameters from checkpoint files, we need to add the paths of checkpoint files in the job configuration file</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">checkpoint_path: PATH_TO_CHECKPOINT_FILE1 +checkpoint_path: PATH_TO_CHECKPOINT_FILE2 +... +</pre></div></div> +<p>The new dataset is configured by specifying the <tt>test_step</tt> and the data input layer, e.g. the following configuration is for a dataset with 100*100 instances.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">test_steps: 100 +net { + layer { + name: "input" + store_conf { + backend: "kvfile" + path: PATH_TO_TEST_KVFILE + batchsize: 100 + } + } + ... +} +</pre></div></div> +<div class="section"> +<h2><a name="Performance_Test"></a>Performance Test</h2> +<p>This application is to test the performance, e.g., accuracy, of the previously trained model. Depending on the application, the test data may have ground truth labels or not. For example, if the model is trained for image classification, the test images must have ground truth labels to calculate the accuracy; if the model is an auto-encoder, the performance could be measured by reconstruction error, which does not require extra labels. For both cases, there would be a layer that calculates the performance, e.g., the <tt>SoftmaxLossLayer</tt>.</p> +<p>The job configuration file for the cifar10 example can be used directly for testing after adding the checkpoint path. The running command is</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">$ ./bin/singa-run.sh -conf examples/cifar10/job.conf -test +</pre></div></div> +<p>The performance would be output on the screen like,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">Load from checkpoint file examples/cifar10/checkpoint/step50000-worker0 +accuracy = 0.728000, loss = 0.807645 +</pre></div></div></div> +<div class="section"> +<h2><a name="Feature_extraction"></a>Feature extraction</h2> +<p>Since deep learning models are good at learning features, feature extraction for is a major functionality of deep learning models, e.g., we can extract features from the fully connected layers of <a href="www.cs.toronto.edu/~fritz/absps/imagenet.pdf">AlexNet</a> as image features for image retrieval. To extract the features from one layer, we simply add an output layer after that layer. For instance, to extract the fully connected (with name <tt>ip1</tt>) layer of the cifar10 example model, we replace the <tt>SoftmaxLossLayer</tt> with a <tt>CSVOutputLayer</tt> which extracts the features into a CSV file,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">layer { + name: "ip1" +} +layer { + name: "output" + type: kCSVOutput + srclayers: "ip1" + store_conf { + backend: "textfile" + path: OUTPUT_FILE_PATH + } +} +</pre></div></div> +<p>The input layer and test steps, and the running command are the same as in <i>Performance Test</i> section.</p></div> +<div class="section"> +<h2><a name="Label_Prediction"></a>Label Prediction</h2> +<p>If the output layer is connected to a layer that predicts labels of images, the output layer would then write the prediction results into files. SINGA provides two built-in layers for generating prediction results, namely,</p> + +<ul> + +<li>SoftmaxLayer, generates probabilities of each candidate labels.</li> + +<li>ArgSortLayer, sorts labels according to probabilities in descending order and keep topk labels.</li> +</ul> +<p>By connecting the two layers with the previous layer and the output layer, we can extract the predictions of each instance. For example,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">layer { + name: "feature" + ... +} +layer { + name: "softmax" + type: kSoftmax + srclayers: "feature" +} +layer { + name: "prediction" + type: kArgSort + srclayers: "softmax" + argsort_conf { + topk: 5 + } +} +layer { + name: "output" + type: kCSVOutput + srclayers: "prediction" + store_conf {} +} +</pre></div></div> +<p>The top-5 labels of each instance will be written as one line of the output CSV file. Currently, above layers cannot co-exist with the loss layers used for training. Please comment out the loss layers for extracting prediction results.</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html>
Added: websites/staging/singa/trunk/content/v0.2.0/train-one-batch.html ============================================================================== --- websites/staging/singa/trunk/content/v0.2.0/train-one-batch.html (added) +++ websites/staging/singa/trunk/content/v0.2.0/train-one-batch.html Tue Apr 12 06:24:50 2016 @@ -0,0 +1,478 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-12 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160412" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – Train-One-Batch</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../docs/index.html" title="English">English</a> +</li> + <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">Train-One-Batch</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>Train-One-Batch</h1> +<hr /> +<p>For each SGD iteration, every worker calls the <tt>TrainOneBatch</tt> function to compute gradients of parameters associated with local layers (i.e., layers dispatched to it). SINGA has implemented two algorithms for the <tt>TrainOneBatch</tt> function. Users select the corresponding algorithm for their model in the configuration.</p> +<div class="section"> +<h2><a name="Basic_user_guide"></a>Basic user guide</h2> +<div class="section"> +<h3><a name="Back-propagation"></a>Back-propagation</h3> +<p><a class="externalLink" href="http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf";>BP algorithm</a> is used for computing gradients of feed-forward models, e.g., <a href="cnn.html">CNN</a> and <a href="mlp.html">MLP</a>, and <a href="rnn.html">RNN</a> models in SINGA.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in job.conf +alg: kBP +</pre></div></div> +<p>To use the BP algorithm for the <tt>TrainOneBatch</tt> function, users just simply configure the <tt>alg</tt> field with <tt>kBP</tt>. If a neural net contains user-defined layers, these layers must be implemented properly be to consistent with the implementation of the BP algorithm in SINGA (see below).</p></div> +<div class="section"> +<h3><a name="Contrastive_Divergence"></a>Contrastive Divergence</h3> +<p><a class="externalLink" href="http://www.cs.toronto.edu/~fritz/absps/nccd.pdf";>CD algorithm</a> is used for computing gradients of energy models like RBM.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># job.conf +alg: kCD +cd_conf { + cd_k: 2 +} +</pre></div></div> +<p>To use the CD algorithm for the <tt>TrainOneBatch</tt> function, users just configure the <tt>alg</tt> field to <tt>kCD</tt>. Uses can also configure the Gibbs sampling steps in the CD algorthm through the <tt>cd_k</tt> field. By default, it is set to 1.</p></div></div> +<div class="section"> +<h2><a name="Advanced_user_guide"></a>Advanced user guide</h2> +<div class="section"> +<h3><a name="Implementation_of_BP"></a>Implementation of BP</h3> +<p>The BP algorithm is implemented in SINGA following the below pseudo code,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">BPTrainOnebatch(step, net) { + // forward propagate + foreach layer in net.local_layers() { + if IsBridgeDstLayer(layer) + recv data from the src layer (i.e., BridgeSrcLayer) + foreach param in layer.params() + Collect(param) // recv response from servers for last update + + layer.ComputeFeature(kForward) + + if IsBridgeSrcLayer(layer) + send layer.data_ to dst layer + } + // backward propagate + foreach layer in reverse(net.local_layers) { + if IsBridgeSrcLayer(layer) + recv gradient from the dst layer (i.e., BridgeDstLayer) + recv response from servers for last update + + layer.ComputeGradient() + foreach param in layer.params() + Update(step, param) // send param.grad_ to servers + + if IsBridgeDstLayer(layer) + send layer.grad_ to src layer + } +} +</pre></div></div> +<p>It forwards features through all local layers (can be checked by layer partition ID and worker ID) and backwards gradients in the reverse order. <a href="layer.html#bridgesrclayer--bridgedstlayer">BridgeSrcLayer</a> (resp. <tt>BridgeDstLayer</tt>) will be blocked until the feature (resp. gradient) from the source (resp. destination) layer comes. Parameter gradients are sent to servers via <tt>Update</tt> function. Updated parameters are collected via <tt>Collect</tt> function, which will be blocked until the parameter is updated. <a href="param.html">Param</a> objects have versions, which can be used to check whether the <tt>Param</tt> objects have been updated or not.</p> +<p>Since RNN models are unrolled into feed-forward models, users need to implement the forward propagation in the recurrent layer’s <tt>ComputeFeature</tt> function, and implement the backward propagation in the recurrent layer’s <tt>ComputeGradient</tt> function. As a result, the whole <tt>TrainOneBatch</tt> runs <a class="externalLink" href="https://en.wikipedia.org/wiki/Backpropagation_through_time";>back-propagation through time (BPTT)</a> algorithm.</p></div> +<div class="section"> +<h3><a name="Implementation_of_CD"></a>Implementation of CD</h3> +<p>The CD algorithm is implemented in SINGA following the below pseudo code,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">CDTrainOneBatch(step, net) { + # positive phase + foreach layer in net.local_layers() + if IsBridgeDstLayer(layer) + recv positive phase data from the src layer (i.e., BridgeSrcLayer) + foreach param in layer.params() + Collect(param) // recv response from servers for last update + layer.ComputeFeature(kPositive) + if IsBridgeSrcLayer(layer) + send positive phase data to dst layer + + # negative phase + foreach gibbs in [0...layer_proto_.cd_k] + foreach layer in net.local_layers() + if IsBridgeDstLayer(layer) + recv negative phase data from the src layer (i.e., BridgeSrcLayer) + layer.ComputeFeature(kPositive) + if IsBridgeSrcLayer(layer) + send negative phase data to dst layer + + foreach layer in net.local_layers() + layer.ComputeGradient() + foreach param in layer.params + Update(param) +} +</pre></div></div> +<p>Parameter gradients are computed after the positive phase and negative phase.</p></div> +<div class="section"> +<h3><a name="Implementing_a_new_algorithm"></a>Implementing a new algorithm</h3> +<p>SINGA implements BP and CD by creating two subclasses of the <a href="../api/classsinga_1_1Worker.html">Worker</a> class: <a href="../api/classsinga_1_1BPWorker.html">BPWorker</a>’s <tt>TrainOneBatch</tt> function implements the BP algorithm; <a href="../api/classsinga_1_1CDWorker.html">CDWorker</a>’s <tt>TrainOneBatch</tt> function implements the CD algorithm. To implement a new algorithm for the <tt>TrainOneBatch</tt> function, users need to create a new subclass of the <tt>Worker</tt>, e.g.,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">class FooWorker : public Worker { + void TrainOneBatch(int step, shared_ptr<NeuralNet> net, Metric* perf) override; + void TestOneBatch(int step, Phase phase, shared_ptr<NeuralNet> net, Metric* perf) override; +}; +</pre></div></div> +<p>The <tt>FooWorker</tt> must implement the above two functions for training one mini-batch and testing one mini-batch. The <tt>perf</tt> argument is for collecting training or testing performance, e.g., the objective loss or accuracy. It is passed to the <tt>ComputeFeature</tt> function of each layer.</p> +<p>Users can define some fields for users to configure</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in user.proto +message FooWorkerProto { + optional int32 b = 1; +} + +extend JobProto { + optional FooWorkerProto foo_conf = 101; +} + +# in job.proto +JobProto { + ... + extension 101..max; +} +</pre></div></div> +<p>It is similar as <a href="layer.html#implementing-a-new-layer-subclass">adding configuration fields for a new layer</a>.</p> +<p>To use <tt>FooWorker</tt>, users need to register it in the <a href="programming-guide.html">main.cc</a> and configure the <tt>alg</tt> and <tt>foo_conf</tt> fields,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in main.cc +const int kFoo = 3; // worker ID, must be different to that of CDWorker and BPWorker +driver.RegisterWorker<FooWorker>(kFoo); + +# in job.conf +... +alg: 3 +[foo_conf] { + b = 4; +} +</pre></div></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html> Added: websites/staging/singa/trunk/content/v0.2.0/updater.html ============================================================================== --- websites/staging/singa/trunk/content/v0.2.0/updater.html (added) +++ websites/staging/singa/trunk/content/v0.2.0/updater.html Tue Apr 12 06:24:50 2016 @@ -0,0 +1,612 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-12 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160412" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – Updater</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../docs/index.html" title="English">English</a> +</li> + <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">Updater</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>Updater</h1> +<hr /> +<p>Every server in SINGA has an <a href="../api/classsinga_1_1Updater.html">Updater</a> instance that updates parameters based on gradients. In this page, the <i>Basic user guide</i> describes the configuration of an updater. The <i>Advanced user guide</i> present details on how to implement a new updater and a new learning rate changing method.</p> +<div class="section"> +<h2><a name="Basic_user_guide"></a>Basic user guide</h2> +<p>There are many different parameter updating protocols (i.e., subclasses of <tt>Updater</tt>). They share some configuration fields like</p> + +<ul> + +<li><tt>type</tt>, an integer for identifying an updater;</li> + +<li><tt>learning_rate</tt>, configuration for the <a href="../api/classsinga_1_1LRGenerator.html">LRGenerator</a> which controls the learning rate.</li> + +<li><tt>weight_decay</tt>, the co-efficient for <a class="externalLink" href="http://deeplearning.net/tutorial/gettingstarted.html#regularization";>L2 * regularization</a>.</li> + +<li><a class="externalLink" href="http://ufldl.stanford.edu/tutorial/supervised/OptimizationStochasticGradientDescent/";>momentum</a>.</li> +</ul> +<p>If you are not familiar with the above terms, you can get their meanings in <a class="externalLink" href="http://cs231n.github.io/neural-networks-3/#update";>this page provided by Karpathy</a>.</p> +<div class="section"> +<h3><a name="Configuration_of_built-in_updater_classes"></a>Configuration of built-in updater classes</h3> +<div class="section"> +<h4><a name="Updater"></a>Updater</h4> +<p>The base <tt>Updater</tt> implements the <a class="externalLink" href="http://cs231n.github.io/neural-networks-3/#sgd";>vanilla SGD algorithm</a>. Its configuration type is <tt>kSGD</tt>. Users need to configure at least the <tt>learning_rate</tt> field. <tt>momentum</tt> and <tt>weight_decay</tt> are optional fields.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">updater{ + type: kSGD + momentum: float + weight_decay: float + learning_rate { + ... + } +} +</pre></div></div></div> +<div class="section"> +<h4><a name="AdaGradUpdater"></a>AdaGradUpdater</h4> +<p>It inherits the base <tt>Updater</tt> to implement the <a class="externalLink" href="http://www.magicbroom.info/Papers/DuchiHaSi10.pdf";>AdaGrad</a> algorithm. Its type is <tt>kAdaGrad</tt>. <tt>AdaGradUpdater</tt> is configured similar to <tt>Updater</tt> except that <tt>momentum</tt> is not used.</p></div> +<div class="section"> +<h4><a name="NesterovUpdater"></a>NesterovUpdater</h4> +<p>It inherits the base <tt>Updater</tt> to implements the <a class="externalLink" href="http://arxiv.org/pdf/1212.0901v2.pdf";>Nesterov</a> (section 3.5) updating protocol. Its type is <tt>kNesterov</tt>. <tt>learning_rate</tt> and <tt>momentum</tt> must be configured. <tt>weight_decay</tt> is an optional configuration field.</p></div> +<div class="section"> +<h4><a name="RMSPropUpdater"></a>RMSPropUpdater</h4> +<p>It inherits the base <tt>Updater</tt> to implements the <a class="externalLink" href="http://cs231n.github.io/neural-networks-3/#sgd";>RMSProp algorithm</a> proposed by <a class="externalLink" href="http://www.cs.toronto.edu/%7Etijmen/csc321/slides/lecture_slides_lec6.pdf";>Hinton</a>(slide 29). Its type is <tt>kRMSProp</tt>.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">updater { + type: kRMSProp + rmsprop_conf { + rho: float # [0,1] + } +} +</pre></div></div></div></div> +<div class="section"> +<h3><a name="Configuration_of_learning_rate"></a>Configuration of learning rate</h3> +<p>The <tt>learning_rate</tt> field is configured as,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + type: ChangeMethod + base_lr: float # base/initial learning rate + ... # fields to a specific changing method +} +</pre></div></div> +<p>The common fields include <tt>type</tt> and <tt>base_lr</tt>. SINGA provides the following <tt>ChangeMethod</tt>s.</p> +<div class="section"> +<h4><a name="kFixed"></a>kFixed</h4> +<p>The <tt>base_lr</tt> is used for all steps.</p></div> +<div class="section"> +<h4><a name="kLinear"></a>kLinear</h4> +<p>The updater should be configured like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + base_lr: float + linear_conf { + freq: int + final_lr: float + } +} +</pre></div></div> +<p>Linear interpolation is used to change the learning rate,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">lr = (1 - step / freq) * base_lr + (step / freq) * final_lr +</pre></div></div></div> +<div class="section"> +<h4><a name="kExponential"></a>kExponential</h4> +<p>The udapter should be configured like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + base_lr: float + exponential_conf { + freq: int + } +} +</pre></div></div> +<p>The learning rate for <tt>step</tt> is</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">lr = base_lr / 2^(step / freq) +</pre></div></div></div> +<div class="section"> +<h4><a name="kInverseT"></a>kInverseT</h4> +<p>The updater should be configured like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + base_lr: float + inverset_conf { + final_lr: float + } +} +</pre></div></div> +<p>The learning rate for <tt>step</tt> is</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">lr = base_lr / (1 + step / final_lr) +</pre></div></div></div> +<div class="section"> +<h4><a name="kInverse"></a>kInverse</h4> +<p>The updater should be configured like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + base_lr: float + inverse_conf { + gamma: float + pow: float + } +} +</pre></div></div> +<p>The learning rate for <tt>step</tt> is</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">lr = base_lr * (1 + gamma * setp)^(-pow) +</pre></div></div></div> +<div class="section"> +<h4><a name="kStep"></a>kStep</h4> +<p>The updater should be configured like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + base_lr : float + step_conf { + change_freq: int + gamma: float + } +} +</pre></div></div> +<p>The learning rate for <tt>step</tt> is</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">lr = base_lr * gamma^ (step / change_freq) +</pre></div></div></div> +<div class="section"> +<h4><a name="kFixedStep"></a>kFixedStep</h4> +<p>The updater should be configured like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + fixedstep_conf { + step: int + step_lr: float + + step: int + step_lr: float + + ... + } +} +</pre></div></div> +<p>Denote the i-th tuple as (step[i], step_lr[i]), then the learning rate for <tt>step</tt> is,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">step_lr[k] +</pre></div></div> +<p>where step[k] is the smallest number that is larger than <tt>step</tt>.</p></div></div></div> +<div class="section"> +<h2><a name="Advanced_user_guide"></a>Advanced user guide</h2> +<div class="section"> +<h3><a name="Implementing_a_new_Updater_subclass"></a>Implementing a new Updater subclass</h3> +<p>The base Updater class has one virtual function,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">class Updater{ + public: + virtual void Update(int step, Param* param, float grad_scale = 1.0f) = 0; + + protected: + UpdaterProto proto_; + LRGenerator lr_gen_; +}; +</pre></div></div> +<p>It updates the values of the <tt>param</tt> based on its gradients. The <tt>step</tt> argument is for deciding the learning rate which may change through time (step). <tt>grad_scale</tt> scales the original gradient values. This function is called by servers once it receives all gradients for the same <tt>Param</tt> object.</p> +<p>To implement a new Updater subclass, users must override the <tt>Update</tt> function.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">class FooUpdater : public Updater { + void Update(int step, Param* param, float grad_scale = 1.0f) override; +}; +</pre></div></div> +<p>Configuration of this new updater can be declared similar to that of a new layer,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in user.proto +FooUpdaterProto { + optional int32 c = 1; +} + +extend UpdaterProto { + optional FooUpdaterProto fooupdater_conf= 101; +} +</pre></div></div> +<p>The new updater should be registered in the <a href="programming-guide.html">main function</a></p> + +<div class="source"> +<div class="source"><pre class="prettyprint">driver.RegisterUpdater<FooUpdater>("FooUpdater"); +</pre></div></div> +<p>Users can then configure the job as</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in job.conf +updater { + user_type: "FooUpdater" # must use user_type with the same string identifier as the one used for registration + fooupdater_conf { + c : 20; + } +} +</pre></div></div></div> +<div class="section"> +<h3><a name="Implementing_a_new_LRGenerator_subclass"></a>Implementing a new LRGenerator subclass</h3> +<p>The base <tt>LRGenerator</tt> is declared as,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">virtual float Get(int step); +</pre></div></div> +<p>To implement a subclass, e.g., <tt>FooLRGen</tt>, users should declare it like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">class FooLRGen : public LRGenerator { + public: + float Get(int step) override; +}; +</pre></div></div> +<p>Configuration of <tt>FooLRGen</tt> can be defined using a protocol message,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in user.proto +message FooLRProto { + ... +} + +extend LRGenProto { + optional FooLRProto foolr_conf = 101; +} +</pre></div></div> +<p>The configuration is then like,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + user_type : "FooLR" # must use user_type with the same string identifier as the one used for registration + base_lr: float + foolr_conf { + ... + } +} +</pre></div></div> +<p>Users have to register this subclass in the main function,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"> driver.RegisterLRGenerator<FooLRGen, std::string>("FooLR") +</pre></div></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html> Added: websites/staging/singa/trunk/content/v0.2.0/zh/checkpoint.html ============================================================================== --- websites/staging/singa/trunk/content/v0.2.0/zh/checkpoint.html (added) +++ websites/staging/singa/trunk/content/v0.2.0/zh/checkpoint.html Tue Apr 12 06:24:50 2016 @@ -0,0 +1,412 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-12 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160412" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – CheckPoint</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../../docs/index.html" title="English">English</a> +</li> + <li> <a href="../../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">CheckPoint</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>CheckPoint</h1> +<hr /> +<p>SINGA checkpoints model parameters onto disk periodically according to user configured frequency. By checkpointing model parameters, we can</p> + +<ol style="list-style-type: decimal"> + +<li> +<p>resume the training from the last checkpointing. For example, if the program crashes before finishing all training steps, we can continue the training using checkpoint files.</p></li> + +<li> +<p>use them to initialize a similar model. For example, the parameters from training a RBM model can be used to initialize a <a href="rbm.html">deep auto-encoder</a> model.</p></li> +</ol> +<div class="section"> +<h2><a name="Configuration"></a>Configuration</h2> +<p>Checkpointing is controlled by two configuration fields:</p> + +<ul> + +<li><tt>checkpoint_after</tt>, start checkpointing after this number of training steps,</li> + +<li><tt>checkpoint_freq</tt>, frequency of doing checkpointing.</li> +</ul> +<p>For example,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># job.conf +checkpoint_after: 100 +checkpoint_frequency: 300 +... +</pre></div></div> +<p>Checkpointing files are located at <i>WORKSPACE/checkpoint/stepSTEP-workerWORKERID</i>. <i>WORKSPACE</i> is configured in</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">cluster { + workspace: +} +</pre></div></div> +<p>For the above configuration, after training for 700 steps, there would be two checkpointing files,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">step400-worker0 +step700-worker0 +</pre></div></div></div> +<div class="section"> +<h2><a name="Application_-_resuming_training"></a>Application - resuming training</h2> +<p>We can resume the training from the last checkpoint (i.e., step 700) by,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">./bin/singa-run.sh -conf JOB_CONF -resume +</pre></div></div> +<p>There is no change to the job configuration.</p></div> +<div class="section"> +<h2><a name="Application_-_model_initialization"></a>Application - model initialization</h2> +<p>We can also use the checkpointing file from step 400 to initialize a new model by configuring the new job as,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># job.conf +checkpoint : "WORKSPACE/checkpoint/step400-worker0" +... +</pre></div></div> +<p>If there are multiple checkpointing files for the same snapshot due to model partitioning, all the checkpointing files should be added,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># job.conf +checkpoint : "WORKSPACE/checkpoint/step400-worker0" +checkpoint : "WORKSPACE/checkpoint/step400-worker1" +... +</pre></div></div> +<p>The training command is the same as starting a new job,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">./bin/singa-run.sh -conf JOB_CONF +</pre></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html>
![https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"](https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"){kind=link}
![http://incubator.apache.org/images/egg-logo.png"](http://incubator.apache.org/images/egg-logo.png"){kind=link}