Added: websites/staging/singa/trunk/content/v0.3.0/rnn.html ============================================================================== --- websites/staging/singa/trunk/content/v0.3.0/rnn.html (added) +++ websites/staging/singa/trunk/content/v0.3.0/rnn.html Wed Apr 20 05:12:03 2016 @@ -0,0 +1,674 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-20 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160420" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – Recurrent Neural Networks for Language Modelling</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../docs/index.html" title="English">English</a> +</li> + <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">Recurrent Neural Networks for Language Modelling</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>Recurrent Neural Networks for Language Modelling</h1> +<hr /> +<p>Recurrent Neural Networks (RNN) are widely used for modelling sequential data, such as music and sentences. In this example, we use SINGA to train a <a class="externalLink" href="http://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf";>RNN model</a> proposed by Tomas Mikolov for <a class="externalLink" href="https://en.wikipedia.org/wiki/Language_model";>language modeling</a>. The training objective (loss) is to minimize the <a class="externalLink" href="https://en.wikipedia.org/wiki/Perplexity";>perplexity per word</a>, which is equivalent to maximize the probability of predicting the next word given the current word in a sentence.</p> +<p>Different to the <a href="cnn.html">CNN</a>, <a href="mlp.html">MLP</a> and <a href="rbm.html">RBM</a> examples which use built-in layers(layer) and records(data), none of the layers in this example are built-in. Hence users would learn to implement their own layers and data records through this example.</p> +<div class="section"> +<h2><a name="Running_instructions"></a>Running instructions</h2> +<p>In <i>SINGA_ROOT/examples/rnnlm/</i>, scripts are provided to run the training job. First, the data is prepared by</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">$ cp Makefile.example Makefile +$ make download +$ make create +</pre></div></div> +<p>Second, to compile the source code under <i>examples/rnnlm/</i>, run</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">$ make rnnlm +</pre></div></div> +<p>An executable file <i>rnnlm.bin</i> will be generated.</p> +<p>Third, the training is started by passing <i>rnnlm.bin</i> and the job configuration to <i>singa-run.sh</i>,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># at SINGA_ROOT/ +# export LD_LIBRARY_PATH=.libs:$LD_LIBRARY_PATH +$ ./bin/singa-run.sh -exec examples/rnnlm/rnnlm.bin -conf examples/rnnlm/job.conf +</pre></div></div></div> +<div class="section"> +<h2><a name="Implementations"></a>Implementations</h2> +<p><img src="../images/rnnlm.png" align="center" width="400px" alt="" /> <span><b>Figure 1 - Net structure of the RNN model.</b></span></p> +<p>The neural net structure is shown Figure 1. Word records are loaded by <tt>DataLayer</tt>. For every iteration, at most <tt>max_window</tt> word records are processed. If a sentence ending character is read, the <tt>DataLayer</tt> stops loading immediately. <tt>EmbeddingLayer</tt> looks up a word embedding matrix to extract feature vectors for words loaded by the <tt>DataLayer</tt>. These features are transformed by the <tt>HiddenLayer</tt> which propagates the features from left to right. The output feature for word at position k is influenced by words from position 0 to k-1. Finally, <tt>LossLayer</tt> computes the cross-entropy loss (see below) by predicting the next word of each word. The cross-entropy loss is computed as</p> +<p><tt>$$L(w_t)=-log P(w_{t+1}|w_t)$$</tt></p> +<p>Given <tt>$w_t$</tt> the above equation would compute over all words in the vocabulary, which is time consuming. <a class="externalLink" href="https://f25ea9ccb7d3346ce6891573d543960492b92c30.googledrive.com/host/0ByxdPXuxLPS5RFM5dVNvWVhTd0U/rnnlm-0.4b.tgz";>RNNLM Toolkit</a> accelerates the computation as</p> +<p><tt>$$P(w_{t+1}|w_t) = P(C_{w_{t+1}}|w_t) * P(w_{t+1}|C_{w_{t+1}})$$</tt></p> +<p>Words from the vocabulary are partitioned into a user-defined number of classes. The first term on the left side predicts the class of the next word, and then predicts the next word given its class. Both the number of classes and the words from one class are much smaller than the vocabulary size. The probabilities can be calculated much faster.</p> +<p>The perplexity per word is computed by,</p> +<p><tt>$$PPL = 10^{- avg_t log_{10} P(w_{t+1}|w_t)}$$</tt></p> +<div class="section"> +<h3><a name="Data_preparation"></a>Data preparation</h3> +<p>We use a small dataset provided by the <a class="externalLink" href="https://f25ea9ccb7d3346ce6891573d543960492b92c30.googledrive.com/host/0ByxdPXuxLPS5RFM5dVNvWVhTd0U/rnnlm-0.4b.tgz";>RNNLM Toolkit</a>. It has 10,000 training sentences, with 71350 words in total and 3720 unique words. The subsequent steps follow the instructions in <a href="data.html">Data Preparation</a> to convert the raw data into records and insert them into data stores.</p> +<div class="section"> +<h4><a name="Download_source_data"></a>Download source data</h4> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in SINGA_ROOT/examples/rnnlm/ +cp Makefile.example Makefile +make download +</pre></div></div></div> +<div class="section"> +<h4><a name="Define_record_format"></a>Define record format</h4> +<p>We define the word record as follows,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in SINGA_ROOT/examples/rnnlm/rnnlm.proto +message WordRecord { + optional string word = 1; + optional int32 word_index = 2; + optional int32 class_index = 3; + optional int32 class_start = 4; + optional int32 class_end = 5; +} +</pre></div></div> +<p>It includes the word string and its index in the vocabulary. Words in the vocabulary are sorted based on their frequency in the training dataset. The sorted list is cut into 100 sublists such that each sublist has 1/100 total word frequency. Each sublist is called a class. Hence each word has a <tt>class_index</tt> ([0,100)). The <tt>class_start</tt> is the index of the first word in the same class as <tt>word</tt>. The <tt>class_end</tt> is the index of the first word in the next class.</p></div> +<div class="section"> +<h4><a name="Create_data_stores"></a>Create data stores</h4> +<p>We use code from RNNLM Toolkit to read words, and sort them into classes. The main function in <i>create_store.cc</i> first creates word classes based on the training dataset. Second it calls the following function to create data store for the training, validation and test dataset.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">int create_data(const char *input_file, const char *output_file); +</pre></div></div> +<p><tt>input</tt> is the path to training/validation/testing text file from the RNNLM Toolkit, <tt>output</tt> is output store file. This function starts with</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">singa::io::KVFile store; +store.Open(output, signa::io::kCreate); +</pre></div></div> +<p>Then it reads the words one by one. For each word it creates a <tt>WordRecord</tt> instance, and inserts it into the store,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">int wcnt = 0; // word count +WordRecord wordRecord; +while(1) { + readWord(wordstr, fin); + if (feof(fin)) break; + ...// fill in the wordRecord; + string val; + wordRecord.SerializeToString(&val); + int length = snprintf(key, BUFFER_LEN, "%05d", wcnt++); + store.Write(string(key, length), val); +} +</pre></div></div> +<p>Compilation and running commands are provided in the <i>Makefile.example</i>. After executing</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">make create +</pre></div></div> +<p><i>train_data.bin</i>, <i>test_data.bin</i> and <i>valid_data.bin</i> will be created.</p></div></div> +<div class="section"> +<h3><a name="Layer_implementation"></a>Layer implementation</h3> +<p>4 user-defined layers are implemented for this application. Following the guide for implementing <a href="layer#implementing-a-new-layer-subclass">new Layer subclasses</a>, we extend the <a href="../api/classsinga_1_1LayerProto.html">LayerProto</a> to include the configuration messages of user-defined layers as shown below (3 out of the 7 layers have specific configurations),</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">import "job.proto"; // Layer message for SINGA is defined + +//For implementation of RNNLM application +extend singa.LayerProto { + optional EmbeddingProto embedding_conf = 101; + optional LossProto loss_conf = 102; + optional DataProto data_conf = 103; +} +</pre></div></div> +<p>In the subsequent sections, we describe the implementation of each layer, including its configuration message.</p> +<div class="section"> +<h4><a name="RNNLayer"></a>RNNLayer</h4> +<p>This is the base layer of all other layers for this applications. It is defined as follows,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">class RNNLayer : virtual public Layer { +public: + inline int window() { return window_; } +protected: + int window_; +}; +</pre></div></div> +<p>For this application, two iterations may process different number of words. Because sentences have different lengths. The <tt>DataLayer</tt> decides the effective window size. All other layers call its source layers to get the effective window size and resets <tt>window_</tt> in <tt>ComputeFeature</tt> function.</p></div> +<div class="section"> +<h4><a name="DataLayer"></a>DataLayer</h4> +<p>DataLayer is for loading Records.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">class DataLayer : public RNNLayer, singa::InputLayer { + public: + void Setup(const LayerProto& proto, const vector<Layer*>& srclayers) override; + void ComputeFeature(int flag, const vector<Layer*>& srclayers) override; + int max_window() const { + return max_window_; + } + private: + int max_window_; + singa::io::Store* store_; +}; +</pre></div></div> +<p>The Setup function gets the user configured max window size.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">max_window_ = proto.GetExtension(input_conf).max_window(); +</pre></div></div> +<p>The <tt>ComputeFeature</tt> function loads at most max_window records. It could also stop when the sentence ending character is encountered.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">...// shift the last record to the first +window_ = max_window_; +for (int i = 1; i <= max_window_; i++) { + // load record; break if it is the ending character +} +</pre></div></div> +<p>The configuration of <tt>DataLayer</tt> is like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">name: "data" +user_type: "kData" +[data_conf] { + path: "examples/rnnlm/train_data.bin" + max_window: 10 +} +</pre></div></div></div> +<div class="section"> +<h4><a name="EmbeddingLayer"></a>EmbeddingLayer</h4> +<p>This layer gets records from <tt>DataLayer</tt>. For each record, the word index is parsed and used to get the corresponding word feature vector from the embedding matrix.</p> +<p>The class is declared as follows,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">class EmbeddingLayer : public RNNLayer { + ... + const std::vector<Param*> GetParams() const override { + std::vector<Param*> params{embed_}; + return params; + } + private: + int word_dim_, vocab_size_; + Param* embed_; +} +</pre></div></div> +<p>The <tt>embed_</tt> field is a matrix whose values are parameter to be learned. The matrix size is <tt>vocab_size_</tt> x <tt>word_dim_</tt>.</p> +<p>The Setup function reads configurations for <tt>word_dim_</tt> and <tt>vocab_size_</tt>. Then it allocates feature Blob for <tt>max_window</tt> words and setups <tt>embed_</tt>.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">int max_window = srclayers[0]->data(this).shape()[0]; +word_dim_ = proto.GetExtension(embedding_conf).word_dim(); +data_.Reshape(vector<int>{max_window, word_dim_}); +... +embed_->Setup(vector<int>{vocab_size_, word_dim_}); +</pre></div></div> +<p>The <tt>ComputeFeature</tt> function simply copies the feature vector from the <tt>embed_</tt> matrix into the feature Blob.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># reset effective window size +window_ = datalayer->window(); +auto records = datalayer->records(); +... +for (int t = 0; t < window_; t++) { + int idx <- word index + Copy(words[t], embed[idx]); +} +</pre></div></div> +<p>The <tt>ComputeGradient</tt> function copies back the gradients to the <tt>embed_</tt> matrix.</p> +<p>The configuration for <tt>EmbeddingLayer</tt> is like,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">user_type: "kEmbedding" +[embedding_conf] { + word_dim: 15 + vocab_size: 3720 +} +srclayers: "data" +param { + name: "w1" + init { + type: kUniform + low:-0.3 + high:0.3 + } +} +</pre></div></div></div> +<div class="section"> +<h4><a name="HiddenLayer"></a>HiddenLayer</h4> +<p>This layer unrolls the recurrent connections for at most max_window times. The feature for position k is computed based on the feature from the embedding layer (position k) and the feature at position k-1 of this layer. The formula is</p> +<p><tt>$$f[k]=\sigma (f[t-1]*W+src[t])$$</tt></p> +<p>where <tt>$W$</tt> is a matrix with <tt>word_dim_</tt> x <tt>word_dim_</tt> parameters.</p> +<p>If you want to implement a recurrent neural network following our design, this layer is of vital importance for you to refer to.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">class HiddenLayer : public RNNLayer { + ... + const std::vector<Param*> GetParams() const override { + std::vector<Param*> params{weight_}; + return params; + } +private: + Param* weight_; +}; +</pre></div></div> +<p>The <tt>Setup</tt> function setups the weight matrix as</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">weight_->Setup(std::vector<int>{word_dim, word_dim}); +</pre></div></div> +<p>The <tt>ComputeFeature</tt> function gets the effective window size (<tt>window_</tt>) from its source layer i.e., the embedding layer. Then it propagates the feature from position 0 to position <tt>window_</tt> -1. The detailed descriptions for this process are illustrated as follows.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">void HiddenLayer::ComputeFeature() { + for(int t = 0; t < window_size; t++){ + if(t == 0) + Copy(data[t], src[t]); + else + data[t]=sigmoid(data[t-1]*W + src[t]); + } +} +</pre></div></div> +<p>The <tt>ComputeGradient</tt> function computes the gradient of the loss w.r.t. W and the source layer. Particularly, for each position k, since data[k] contributes to data[k+1] and the feature at position k in its destination layer (the loss layer), grad[k] should contains the gradient from two parts. The destination layer has already computed the gradient from the loss layer into grad[k]; In the <tt>ComputeGradient</tt> function, we need to add the gradient from position k+1.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">void HiddenLayer::ComputeGradient(){ + ... + for (int k = window_ - 1; k >= 0; k--) { + if (k < window_ - 1) { + grad[k] += dot(grad[k + 1], weight.T()); // add gradient from position t+1. + } + grad[k] =... // compute gL/gy[t], y[t]=data[t-1]*W+src[t] + } + gweight = dot(data.Slice(0, window_-1).T(), grad.Slice(1, window_)); + Copy(gsrc, grad); +} +</pre></div></div> +<p>After the loop, we get the gradient of the loss w.r.t y[k], which is used to compute the gradient of W and the src[k].</p></div> +<div class="section"> +<h4><a name="LossLayer"></a>LossLayer</h4> +<p>This layer computes the cross-entropy loss and the <tt>$log_{10}P(w_{t+1}|w_t)$</tt> (which could be averaged over all words by users to get the PPL value).</p> +<p>There are two configuration fields to be specified by users.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">message LossProto { + optional int32 nclass = 1; + optional int32 vocab_size = 2; +} +</pre></div></div> +<p>There are two weight matrices to be learned</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">class LossLayer : public RNNLayer { + ... + private: + Param* word_weight_, *class_weight_; +} +</pre></div></div> +<p>The ComputeFeature function computes the two probabilities respectively.</p> +<p><tt>$$P(C_{w_{t+1}}|w_t) = Softmax(w_t * class\_weight_)$$</tt> <tt>$$P(w_{t+1}|C_{w_{t+1}}) = Softmax(w_t * word\_weight[class\_start:class\_end])$$</tt></p> +<p><tt>$w_t$</tt> is the feature from the hidden layer for the k-th word, its ground truth next word is <tt>$w_{t+1}$</tt>. The first equation computes the probability distribution over all classes for the next word. The second equation computes the probability distribution over the words in the ground truth class for the next word.</p> +<p>The ComputeGradient function computes the gradient of the source layer (i.e., the hidden layer) and the two weight matrices.</p></div></div> +<div class="section"> +<h3><a name="Updater_Configuration"></a>Updater Configuration</h3> +<p>We employ kFixedStep type of the learning rate change method and the configuration is as follows. We decay the learning rate once the performance does not increase on the validation dataset.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">updater{ + type: kSGD + learning_rate { + type: kFixedStep + fixedstep_conf:{ + step:0 + step:48810 + step:56945 + step:65080 + step:73215 + step_lr:0.1 + step_lr:0.05 + step_lr:0.025 + step_lr:0.0125 + step_lr:0.00625 + } + } +} +</pre></div></div></div> +<div class="section"> +<h3><a name="TrainOneBatch_Function"></a>TrainOneBatch() Function</h3> +<p>We use BP (BackPropagation) algorithm to train the RNN model here. The corresponding configuration can be seen below.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># In job.conf file +train_one_batch { + alg: kBackPropagation +} +</pre></div></div></div> +<div class="section"> +<h3><a name="Cluster_Configuration"></a>Cluster Configuration</h3> +<p>The default cluster configuration can be used, i.e., single worker and single server in a single process.</p></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html>
Added: websites/staging/singa/trunk/content/v0.3.0/test.html ============================================================================== --- websites/staging/singa/trunk/content/v0.3.0/test.html (added) +++ websites/staging/singa/trunk/content/v0.3.0/test.html Wed Apr 20 05:12:03 2016 @@ -0,0 +1,436 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-20 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160420" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – Performance Test and Feature Extraction</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../docs/index.html" title="English">English</a> +</li> + <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">Performance Test and Feature Extraction</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>Performance Test and Feature Extraction</h1> +<hr /> +<p>Once SINGA finishes the training of a model, it would checkpoint the model parameters into disk files under the <a href="checkpoint.html">checkpoint folder</a>. Model parameters can also be dumped into this folder periodically during training if the [checkpoint configuration[(checkpoint.html) fields are set. With the checkpoint files, we can load the model parameters to conduct performance test, feature extraction and prediction against new data.</p> +<p>To load the model parameters from checkpoint files, we need to add the paths of checkpoint files in the job configuration file</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">checkpoint_path: PATH_TO_CHECKPOINT_FILE1 +checkpoint_path: PATH_TO_CHECKPOINT_FILE2 +... +</pre></div></div> +<p>The new dataset is configured by specifying the <tt>test_step</tt> and the data input layer, e.g. the following configuration is for a dataset with 100*100 instances.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">test_steps: 100 +net { + layer { + name: "input" + store_conf { + backend: "kvfile" + path: PATH_TO_TEST_KVFILE + batchsize: 100 + } + } + ... +} +</pre></div></div> +<div class="section"> +<h2><a name="Performance_Test"></a>Performance Test</h2> +<p>This application is to test the performance, e.g., accuracy, of the previously trained model. Depending on the application, the test data may have ground truth labels or not. For example, if the model is trained for image classification, the test images must have ground truth labels to calculate the accuracy; if the model is an auto-encoder, the performance could be measured by reconstruction error, which does not require extra labels. For both cases, there would be a layer that calculates the performance, e.g., the <tt>SoftmaxLossLayer</tt>.</p> +<p>The job configuration file for the cifar10 example can be used directly for testing after adding the checkpoint path. The running command is</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">$ ./bin/singa-run.sh -conf examples/cifar10/job.conf -test +</pre></div></div> +<p>The performance would be output on the screen like,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">Load from checkpoint file examples/cifar10/checkpoint/step50000-worker0 +accuracy = 0.728000, loss = 0.807645 +</pre></div></div></div> +<div class="section"> +<h2><a name="Feature_extraction"></a>Feature extraction</h2> +<p>Since deep learning models are good at learning features, feature extraction for is a major functionality of deep learning models, e.g., we can extract features from the fully connected layers of <a href="www.cs.toronto.edu/~fritz/absps/imagenet.pdf">AlexNet</a> as image features for image retrieval. To extract the features from one layer, we simply add an output layer after that layer. For instance, to extract the fully connected (with name <tt>ip1</tt>) layer of the cifar10 example model, we replace the <tt>SoftmaxLossLayer</tt> with a <tt>CSVOutputLayer</tt> which extracts the features into a CSV file,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">layer { + name: "ip1" +} +layer { + name: "output" + type: kCSVOutput + srclayers: "ip1" + store_conf { + backend: "textfile" + path: OUTPUT_FILE_PATH + } +} +</pre></div></div> +<p>The input layer and test steps, and the running command are the same as in <i>Performance Test</i> section.</p></div> +<div class="section"> +<h2><a name="Label_Prediction"></a>Label Prediction</h2> +<p>If the output layer is connected to a layer that predicts labels of images, the output layer would then write the prediction results into files. SINGA provides two built-in layers for generating prediction results, namely,</p> + +<ul> + +<li>SoftmaxLayer, generates probabilities of each candidate labels.</li> + +<li>ArgSortLayer, sorts labels according to probabilities in descending order and keep topk labels.</li> +</ul> +<p>By connecting the two layers with the previous layer and the output layer, we can extract the predictions of each instance. For example,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">layer { + name: "feature" + ... +} +layer { + name: "softmax" + type: kSoftmax + srclayers: "feature" +} +layer { + name: "prediction" + type: kArgSort + srclayers: "softmax" + argsort_conf { + topk: 5 + } +} +layer { + name: "output" + type: kCSVOutput + srclayers: "prediction" + store_conf {} +} +</pre></div></div> +<p>The top-5 labels of each instance will be written as one line of the output CSV file. Currently, above layers cannot co-exist with the loss layers used for training. Please comment out the loss layers for extracting prediction results.</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html> Added: websites/staging/singa/trunk/content/v0.3.0/train-one-batch.html ============================================================================== --- websites/staging/singa/trunk/content/v0.3.0/train-one-batch.html (added) +++ websites/staging/singa/trunk/content/v0.3.0/train-one-batch.html Wed Apr 20 05:12:03 2016 @@ -0,0 +1,478 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-20 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160420" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – Train-One-Batch</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../docs/index.html" title="English">English</a> +</li> + <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">Train-One-Batch</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>Train-One-Batch</h1> +<hr /> +<p>For each SGD iteration, every worker calls the <tt>TrainOneBatch</tt> function to compute gradients of parameters associated with local layers (i.e., layers dispatched to it). SINGA has implemented two algorithms for the <tt>TrainOneBatch</tt> function. Users select the corresponding algorithm for their model in the configuration.</p> +<div class="section"> +<h2><a name="Basic_user_guide"></a>Basic user guide</h2> +<div class="section"> +<h3><a name="Back-propagation"></a>Back-propagation</h3> +<p><a class="externalLink" href="http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf";>BP algorithm</a> is used for computing gradients of feed-forward models, e.g., <a href="cnn.html">CNN</a> and <a href="mlp.html">MLP</a>, and <a href="rnn.html">RNN</a> models in SINGA.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in job.conf +alg: kBP +</pre></div></div> +<p>To use the BP algorithm for the <tt>TrainOneBatch</tt> function, users just simply configure the <tt>alg</tt> field with <tt>kBP</tt>. If a neural net contains user-defined layers, these layers must be implemented properly be to consistent with the implementation of the BP algorithm in SINGA (see below).</p></div> +<div class="section"> +<h3><a name="Contrastive_Divergence"></a>Contrastive Divergence</h3> +<p><a class="externalLink" href="http://www.cs.toronto.edu/~fritz/absps/nccd.pdf";>CD algorithm</a> is used for computing gradients of energy models like RBM.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># job.conf +alg: kCD +cd_conf { + cd_k: 2 +} +</pre></div></div> +<p>To use the CD algorithm for the <tt>TrainOneBatch</tt> function, users just configure the <tt>alg</tt> field to <tt>kCD</tt>. Uses can also configure the Gibbs sampling steps in the CD algorthm through the <tt>cd_k</tt> field. By default, it is set to 1.</p></div></div> +<div class="section"> +<h2><a name="Advanced_user_guide"></a>Advanced user guide</h2> +<div class="section"> +<h3><a name="Implementation_of_BP"></a>Implementation of BP</h3> +<p>The BP algorithm is implemented in SINGA following the below pseudo code,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">BPTrainOnebatch(step, net) { + // forward propagate + foreach layer in net.local_layers() { + if IsBridgeDstLayer(layer) + recv data from the src layer (i.e., BridgeSrcLayer) + foreach param in layer.params() + Collect(param) // recv response from servers for last update + + layer.ComputeFeature(kForward) + + if IsBridgeSrcLayer(layer) + send layer.data_ to dst layer + } + // backward propagate + foreach layer in reverse(net.local_layers) { + if IsBridgeSrcLayer(layer) + recv gradient from the dst layer (i.e., BridgeDstLayer) + recv response from servers for last update + + layer.ComputeGradient() + foreach param in layer.params() + Update(step, param) // send param.grad_ to servers + + if IsBridgeDstLayer(layer) + send layer.grad_ to src layer + } +} +</pre></div></div> +<p>It forwards features through all local layers (can be checked by layer partition ID and worker ID) and backwards gradients in the reverse order. <a href="layer.html#bridgesrclayer--bridgedstlayer">BridgeSrcLayer</a> (resp. <tt>BridgeDstLayer</tt>) will be blocked until the feature (resp. gradient) from the source (resp. destination) layer comes. Parameter gradients are sent to servers via <tt>Update</tt> function. Updated parameters are collected via <tt>Collect</tt> function, which will be blocked until the parameter is updated. <a href="param.html">Param</a> objects have versions, which can be used to check whether the <tt>Param</tt> objects have been updated or not.</p> +<p>Since RNN models are unrolled into feed-forward models, users need to implement the forward propagation in the recurrent layer’s <tt>ComputeFeature</tt> function, and implement the backward propagation in the recurrent layer’s <tt>ComputeGradient</tt> function. As a result, the whole <tt>TrainOneBatch</tt> runs <a class="externalLink" href="https://en.wikipedia.org/wiki/Backpropagation_through_time";>back-propagation through time (BPTT)</a> algorithm.</p></div> +<div class="section"> +<h3><a name="Implementation_of_CD"></a>Implementation of CD</h3> +<p>The CD algorithm is implemented in SINGA following the below pseudo code,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">CDTrainOneBatch(step, net) { + # positive phase + foreach layer in net.local_layers() + if IsBridgeDstLayer(layer) + recv positive phase data from the src layer (i.e., BridgeSrcLayer) + foreach param in layer.params() + Collect(param) // recv response from servers for last update + layer.ComputeFeature(kPositive) + if IsBridgeSrcLayer(layer) + send positive phase data to dst layer + + # negative phase + foreach gibbs in [0...layer_proto_.cd_k] + foreach layer in net.local_layers() + if IsBridgeDstLayer(layer) + recv negative phase data from the src layer (i.e., BridgeSrcLayer) + layer.ComputeFeature(kPositive) + if IsBridgeSrcLayer(layer) + send negative phase data to dst layer + + foreach layer in net.local_layers() + layer.ComputeGradient() + foreach param in layer.params + Update(param) +} +</pre></div></div> +<p>Parameter gradients are computed after the positive phase and negative phase.</p></div> +<div class="section"> +<h3><a name="Implementing_a_new_algorithm"></a>Implementing a new algorithm</h3> +<p>SINGA implements BP and CD by creating two subclasses of the <a href="../api/classsinga_1_1Worker.html">Worker</a> class: <a href="../api/classsinga_1_1BPWorker.html">BPWorker</a>’s <tt>TrainOneBatch</tt> function implements the BP algorithm; <a href="../api/classsinga_1_1CDWorker.html">CDWorker</a>’s <tt>TrainOneBatch</tt> function implements the CD algorithm. To implement a new algorithm for the <tt>TrainOneBatch</tt> function, users need to create a new subclass of the <tt>Worker</tt>, e.g.,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">class FooWorker : public Worker { + void TrainOneBatch(int step, shared_ptr<NeuralNet> net, Metric* perf) override; + void TestOneBatch(int step, Phase phase, shared_ptr<NeuralNet> net, Metric* perf) override; +}; +</pre></div></div> +<p>The <tt>FooWorker</tt> must implement the above two functions for training one mini-batch and testing one mini-batch. The <tt>perf</tt> argument is for collecting training or testing performance, e.g., the objective loss or accuracy. It is passed to the <tt>ComputeFeature</tt> function of each layer.</p> +<p>Users can define some fields for users to configure</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in user.proto +message FooWorkerProto { + optional int32 b = 1; +} + +extend JobProto { + optional FooWorkerProto foo_conf = 101; +} + +# in job.proto +JobProto { + ... + extension 101..max; +} +</pre></div></div> +<p>It is similar as <a href="layer.html#implementing-a-new-layer-subclass">adding configuration fields for a new layer</a>.</p> +<p>To use <tt>FooWorker</tt>, users need to register it in the <a href="programming-guide.html">main.cc</a> and configure the <tt>alg</tt> and <tt>foo_conf</tt> fields,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in main.cc +const int kFoo = 3; // worker ID, must be different to that of CDWorker and BPWorker +driver.RegisterWorker<FooWorker>(kFoo); + +# in job.conf +... +alg: 3 +[foo_conf] { + b = 4; +} +</pre></div></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html>
![https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"](https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"){kind=link}
![http://incubator.apache.org/images/egg-logo.png"](http://incubator.apache.org/images/egg-logo.png"){kind=link}