Added: websites/staging/singa/trunk/content/v0.3.0/python_interactive_training.html ============================================================================== --- websites/staging/singa/trunk/content/v0.3.0/python_interactive_training.html (added) +++ websites/staging/singa/trunk/content/v0.3.0/python_interactive_training.html Wed Apr 20 05:12:03 2016 @@ -0,0 +1,512 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-20 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160420" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – Interactive Training using Python</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../docs/index.html" title="English">English</a> +</li> + <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">Interactive Training using Python</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>Interactive Training using Python</h1> +<hr /> +<p><tt>Layer</tt> class (<a href="layer.py">layer.py</a>) has the following methods for an interactive training. For the basic usage of Python binding features, please refer to <a href="python.md">python.md</a>.</p> +<p><b>ComputeFeature(self, *srclys)</b></p> + +<ul> + +<li> +<p>This method creates and sets up singa::Layer and maintains its source layers, then call singa::Layer::ComputeFeature(…) for data transformation.</p> + +<ul> + +<li><tt>*srclys</tt>: (an arbtrary number of) source layers</li> + </ul></li> +</ul> +<p><b>ComputeGradient(self)</b></p> + +<ul> + +<li>This method creates calls singa::Layer::ComputeGradient(…) for gradient computation.</li> +</ul> +<p><b>GetParams(self)</b></p> + +<ul> + +<li>This method calls singa::Layer::GetParam() to retrieve parameter values of the layer. Currently, it returns weight and bias. Each parameter is a 2D numpy array.</li> +</ul> +<p><b>SetParams(self, *params)</b></p> + +<ul> + +<li>This method sets parameter values of the layer. + +<ul> + +<li><tt>*params</tt>: (an arbitrary number of) parameters, each of which is a 2D numpy array. Typically, it sets weight and bias, 2D numpy array.</li> + </ul></li> +</ul> +<hr /> +<p><tt>Dummy</tt> class is a subclass of <tt>Layer</tt>, which is provided to fetch input data and/or label information. Specifically, it creates singa::DummyLayer.</p> +<p><b>Feed(self, shape, data, aux_data)</b></p> + +<ul> + +<li> +<p>This method sets input data and/or auxiary data such as labels.</p> + +<ul> + +<li><tt>shape</tt>: the shape (width and height) of dataset</li> + +<li><tt>data</tt>: input dataset</li> + +<li><tt>aux_data</tt>: auxiary dataset (e.g., labels)</li> + </ul></li> +</ul> +<p>In addition, <tt>Dummy</tt> class has two subclasses named <tt>ImageInput</tt> and <tt>LabelInput</tt>.</p> + +<ul> + +<li> +<p><tt>ImageInput</tt> class will take three arguments as follows.</p> +<p><b>__init__(self, height=None, width=None, nb_channel=1)</b></p></li> + +<li> +<p>Both <tt>ImageInput</tt> and <tt>LabelInput</tt> classes have their own Feed method to call Feed of Dummy class.</p> +<p><b>Feed(self, data)</b></p></li> +</ul> +<!-- Users can save or load model parameter (e.g., weight and bias) at anytime during training. +The following methods are provided in `model.py`. + +**save_model_parameter(step, fout, neuralnet)** + +* This method saves model parameters into the specified checkpoint (fout). + + * `step`: the step id of training + * `fout`: the name of checkpoint (output filename) + * `neuralnet`: neural network model, i.e., a list of layers + +**load_model_parameter(fin, neuralnet, batchsize=1, data_shape=None)** + +* This method loads model parameters from the specified checkpoint (fin). + + * `fin`: the name of checkpoint (input filename) + * `neuralnet`: neural network model, i.e., a list of layers + * `batchsize`: + * `data_shape`: --> +<hr /> +<div class="section"> +<h2><a name="Example_scripts_for_the_interactive_training"></a>Example scripts for the interactive training</h2> +<p>Two example scripts are provided at <a href=""><tt>train_mnist.py</tt></a> and <a href=""><tt>train_cifar10.py</tt></a>, one is training MLP model for MNIST dataset, and another is training CNN model for CIFAR10 dataset.</p> + +<ul> + +<li> +<p>Assume that <tt>nn</tt> is a neural network model, i.e., a list of layers. Currently, this examples considers sequential models. Example MLP and CNN are shown below.</p></li> + +<li> +<p><tt>load_dataset()</tt> method loads input data and corresponding labels, each of which is a 2D numpy array. For example, loading MNIST dataset returns x: [60000 x 784] and y: [60000 x 1]. Loading CIFAR10 dataset, x: [10000 x 3072] and y: [10000 x 1].</p></li> + +<li> +<p><tt>sgd</tt> is an Updater instance. Please see <a href="python.md"><tt>python.md</tt></a> and <a href=""><tt>model.py</tt></a> for more details.</p></li> +</ul> +<div class="section"> +<div class="section"> +<h4><a name="Basic_steps_for_the_interactive_training"></a>Basic steps for the interactive training</h4> + +<ul> + +<li> +<p>Step 1: Prepare batchsized data and corresponding label information, and then input the data using <tt>Feed()</tt> method.</p></li> + +<li> +<p>Step 2: (a) Transform data according to neuralnet (nn) structure using <tt>ComputeFeature()</tt>. Note that this example considers a sequential model, so it uses a simple loop. (b) Users need to provide <tt>label</tt> information for loss layer to compute loss function. (c) Users can print out the training performance, e.g., loss and accuracy.</p></li> + +<li> +<p>Step 3: Compute gradient in a reverse order of neuralnet (nn) structure using <tt>ComputeGradient()</tt>.</p></li> + +<li> +<p>Step 4: Update parameters, e.g., weight and bias, of layers using <tt>Update()</tt> of the updater.</p></li> +</ul> +<p>Here is an example script for the interactive training. ``` bsize = 64 # batchsize disp_freq = 10 # step to show the training accuracy</p> +<p>x, y = load_dataset()</p> +<p>for i in range(x.shape[0] / bsize):</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># (Step1) Input data containing "bsize" samples +xb, yb = x[i*bsize:(i+1)*bsize, :], y[i*bsize:(i+1)*bsize, :] +nn[0].Feed(xb) +label.Feed(yb) + +# (Step2-a) Transform data according to the neuralnet (nn) structure +for h in range(1, len(nn)): + nn[h].ComputeFeature(nn[h-1]) + +# (Step2-b) Provide label to compute loss function +loss.ComputeFeature(nn[-1], label) + +# (Step2-c) Print out performance, e.g., loss and accuracy +if (i+1) % disp_freq == 0: + print ' Step {:>3}: '.format(i+1), + loss.display() + +# (Step3) Compute gradient in a reverse order +loss.ComputeGradient() +for h in range(len(nn)-1, 0, -1): + nn[h].ComputeGradient() + # (Step 4) Update parameter + sgd.Update(i+1, nn[h]) +</pre></div></div> + +<div class="source"> +<div class="source"><pre class="prettyprint"><a id="model"></a> +### <a href="#model">Example MLP</a> + +Here is an example MLP model with 5 fully-connected hidden layers. +Please refer to [`python.md`](python.md) and [`layer.py`]() for more details about layer definition. `SGD()` is an updater defined in [`model.py`](). + +</pre></div></div> +<p>input = ImageInput(28, 28) # image width and height label = LabelInput()</p> +<p>nn = [] nn.append(input) nn.append(Dense(2500, init=‘uniform’)) nn.append(Activation(‘stanh’)) nn.append(Dense(2000, init=‘uniform’)) nn.append(Activation(‘stanh’)) nn.append(Dense(1500, init=‘uniform’)) nn.append(Activation(‘stanh’)) nn.append(Dense(1000, init=‘uniform’)) nn.append(Activation(‘stanh’)) nn.append(Dense(500, init=‘uniform’)) nn.append(Activation(‘stanh’)) nn.append(Dense(10, init=‘uniform’)) loss = Loss(‘softmaxloss’)</p> +<p>sgd = SGD(lr=0.001, lr_type=‘step’)</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">### <a href="#model2">Example CNN</a> + +Here is an example MLP model with 3 convolution and pooling layers. +Please refer to [`python.md`]() and [`layer.py`]() for more details about layer definition. `SGD()` is an updater defined in [`model.py`](). + +</pre></div></div> +<p>input = ImageInput(32, 32, 3) # image width, height, channel label = LabelInput()</p> +<p>nn = [] nn.append(input) nn.append(Convolution2D(32, 5, 1, 2, w_std=0.0001, b_lr=2)) nn.append(MaxPooling2D(pool_size=(3,3), stride=2)) nn.append(Activation(‘relu’)) nn.append(LRN2D(3, alpha=0.00005, beta=0.75)) nn.append(Convolution2D(32, 5, 1, 2, b_lr=2)) nn.append(Activation(‘relu’)) nn.append(AvgPooling2D(pool_size=(3,3), stride=2)) nn.append(LRN2D(3, alpha=0.00005, beta=0.75)) nn.append(Convolution2D(64, 5, 1, 2)) nn.append(Activation(‘relu’)) nn.append(AvgPooling2D(pool_size=(3,3), stride=2)) nn.append(Dense(10, w_wd=250, b_lr=2, b_wd=0)) loss = Loss(‘softmaxloss’)</p> +<p>sgd = SGD(decay=0.004, momentum=0.9, lr_type=‘manual’, step=(0,60000,65000), step_lr=(0.001,0.0001,0.00001)) ```</p></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html>
Added: websites/staging/singa/trunk/content/v0.3.0/quick-start.html ============================================================================== --- websites/staging/singa/trunk/content/v0.3.0/quick-start.html (added) +++ websites/staging/singa/trunk/content/v0.3.0/quick-start.html Wed Apr 20 05:12:03 2016 @@ -0,0 +1,479 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-20 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160420" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – Quick Start</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../docs/index.html" title="English">English</a> +</li> + <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">Quick Start</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>Quick Start</h1> +<hr /> +<div class="section"> +<h2><a name="SINGA_setup"></a>SINGA setup</h2> +<p>Please refer to the <a href="installation.html">installation</a> page for guidance on installing SINGA.</p> +<div class="section"> +<h3><a name="Training_on_a_single_node"></a>Training on a single node</h3> +<p>For single node training, one process will be launched to run SINGA at local host. We train the <a class="externalLink" href="http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks";>CNN model</a> over the <a class="externalLink" href="http://www.cs.toronto.edu/~kriz/cifar.html";>CIFAR-10</a> dataset as an example. The hyper-parameters are set following <a class="externalLink" href="https://code.google.com/p/cuda-convnet/";>cuda-convnet</a>. More details is available at <a href="cnn.html">CNN example</a>.</p> +<div class="section"> +<h4><a name="Preparing_data_and_job_configuration"></a>Preparing data and job configuration</h4> +<p>Download the dataset and create the data shards for training and testing.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">cd examples/cifar10/ +cp Makefile.example Makefile +make download +make create +</pre></div></div> +<p>A training dataset and a test dataset are created respectively. An <i>image_mean.bin</i> file is also generated, which contains the feature mean of all images.</p> +<p>Since all code used for training this CNN model is provided by SINGA as built-in implementation, there is no need to write any code. Instead, users just execute the running script by providing the job configuration file (<i>job.conf</i>). To code in SINGA, please refer to the <a href="programming-guide.html">programming guide</a>.</p></div> +<div class="section"> +<h4><a name="Training_without_parallelism"></a>Training without parallelism</h4> +<p>By default, the cluster topology has a single worker and a single server. In other words, neither the training data nor the neural net is partitioned.</p> +<p>The training is started by running:</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># goto top level folder +cd ../../ +./singa -conf examples/cifar10/job.conf +</pre></div></div></div> +<div class="section"> +<h4><a name="Asynchronous_parallel_training"></a>Asynchronous parallel training</h4> + +<div class="source"> +<div class="source"><pre class="prettyprint"># job.conf +... +cluster { + nworker_groups: 2 + nworkers_per_procs: 2 + workspace: "examples/cifar10/" +} +</pre></div></div> +<p>In SINGA, <a href="architecture.html">asynchronous training</a> is enabled by launching multiple worker groups. For example, we can change the original <i>job.conf</i> to have two worker groups as shown above. By default, each worker group has one worker. Since one process is set to contain two workers. The two worker groups will run in the same process. Consequently, they run the in-memory <a href="frameworks.html">Downpour</a> training framework. Users do not need to split the dataset explicitly for each worker (group); instead, they can assign each worker (group) a random offset to the start of the dataset. The workers would run as on different data partitions.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># job.conf +... +neuralnet { + layer { + ... + store_conf { + random_skip: 5000 + } + } + ... +} +</pre></div></div> +<p>The running command is:</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">./singa -conf examples/cifar10/job.conf +</pre></div></div></div> +<div class="section"> +<h4><a name="Synchronous_parallel_training"></a>Synchronous parallel training</h4> + +<div class="source"> +<div class="source"><pre class="prettyprint"># job.conf +... +cluster { + nworkers_per_group: 2 + nworkers_per_procs: 2 + workspace: "examples/cifar10/" +} +</pre></div></div> +<p>In SINGA, <a href="architecture.html">asynchronous training</a> is enabled by launching multiple workers within one worker group. For instance, we can change the original <i>job.conf</i> to have two workers in one worker group as shown above. The workers will run synchronously as they are from the same worker group. This framework is the in-memory <a href="frameworks.html">sandblaster</a>. The model is partitioned among the two workers. In specific, each layer is sliced over the two workers. The sliced layer is the same as the original layer except that it only has <tt>B/g</tt> feature instances, where <tt>B</tt> is the number of instances in a mini-batch, <tt>g</tt> is the number of workers in a group. It is also possible to partition the layer (or neural net) using <a href="neural-net.html">other schemes</a>. All other settings are the same as running without partitioning</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">./singa -conf examples/cifar10/job.conf +</pre></div></div></div></div> +<div class="section"> +<h3><a name="Training_in_a_cluster"></a>Training in a cluster</h3> +<div class="section"> +<h4><a name="Starting_Zookeeper"></a>Starting Zookeeper</h4> +<p>SINGA uses <a class="externalLink" href="https://zookeeper.apache.org/";>zookeeper</a> to coordinate the training, and uses ZeroMQ for transferring messages. After installing zookeeper and ZeroMQ, you need to configure SINGA with <tt>--enable-dist</tt> before compiling. Please make sure the zookeeper service is started before running SINGA.</p> +<p>If you installed the zookeeper using our thirdparty script, you can simply start it by:</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">#goto top level folder +cd SINGA_ROOT +./bin/zk-service.sh start +</pre></div></div> +<p>(<tt>./bin/zk-service.sh stop</tt> stops the zookeeper).</p> +<p>Otherwise, if you launched a zookeeper by yourself but not used the default port, please edit the <tt>conf/singa.conf</tt>:</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">zookeeper_host: "localhost:YOUR_PORT" +</pre></div></div> +<p>We can extend the above two training frameworks to a cluster by updating the cluster configuration with:</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">nworker_per_procs: 1 +</pre></div></div> +<p>Every process would then create only one worker thread. Consequently, the workers would be created in different processes (i.e., nodes). The <i>hostfile</i> must be provided under <i>SINGA_ROOT/conf/</i> specifying the nodes in the cluster, e.g.,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">192.168.0.1 +192.168.0.2 +</pre></div></div> +<p>And the zookeeper location must be configured correctly, e.g.,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">#conf/singa.conf +zookeeper_host: "logbase-a01" +</pre></div></div> +<p>The running command is :</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">./bin/singa-run.sh -conf examples/cifar10/job.conf +</pre></div></div> +<p>You can list the current running jobs by,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">./bin/singa-console.sh list + +JOB ID |NUM PROCS +----------|----------- +24 |2 +</pre></div></div> +<p>Jobs can be killed by,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">./bin/singa-console.sh kill JOB_ID +</pre></div></div> +<p>Logs and job information are available in <i>/tmp/singa-log</i> folder, which can be changed to other folders by setting <tt>log-dir</tt> in <i>conf/singa.conf</i>.</p></div></div> +<div class="section"> +<h3><a name="Training_with_GPUs"></a>Training with GPUs</h3> +<p>Please refer to the [GPU page][gpu.html] for details on training using GPUs.</p></div></div> +<div class="section"> +<h2><a name="Where_to_go_next"></a>Where to go next</h2> +<p>The <a href="programming-guide.html">programming guide</a> pages will describe how to submit a training job in SINGA.</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html> Added: websites/staging/singa/trunk/content/v0.3.0/rbm.html ============================================================================== --- websites/staging/singa/trunk/content/v0.3.0/rbm.html (added) +++ websites/staging/singa/trunk/content/v0.3.0/rbm.html Wed Apr 20 05:12:03 2016 @@ -0,0 +1,643 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-20 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160420" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – RBM Example</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../docs/index.html" title="English">English</a> +</li> + <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">RBM Example</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>RBM Example</h1> +<hr /> +<p>This example uses SINGA to train 4 RBM models and one auto-encoder model over the <a class="externalLink" href="http://yann.lecun.com/exdb/mnist/";>MNIST dataset</a>. The auto-encoder model is trained to reduce the dimensionality of the MNIST image feature. The RBM models are trained to initialize parameters of the auto-encoder model. This example application is from <a class="externalLink" href="http://www.cs.toronto.edu/~hinton/science.pdf";>Hinton’s science paper</a>.</p> +<div class="section"> +<h2><a name="Running_instructions"></a>Running instructions</h2> +<p>Running scripts are provided in <i>SINGA_ROOT/examples/rbm</i> folder.</p> +<p>The MNIST dataset has 70,000 handwritten digit images. The <a href="data.html">data preparation</a> page has details on converting this dataset into SINGA recognizable format. Users can simply run the following commands to download and convert the dataset.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># at SINGA_ROOT/examples/mnist/ +$ cp Makefile.example Makefile +$ make download +$ make create +</pre></div></div> +<p>The training is separated into two phases, namely pre-training and fine-tuning. The pre-training phase trains 4 RBMs in sequence,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># at SINGA_ROOT/ +$ ./bin/singa-run.sh -conf examples/rbm/rbm1.conf +$ ./bin/singa-run.sh -conf examples/rbm/rbm2.conf +$ ./bin/singa-run.sh -conf examples/rbm/rbm3.conf +$ ./bin/singa-run.sh -conf examples/rbm/rbm4.conf +</pre></div></div> +<p>The fine-tuning phase trains the auto-encoder by,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">$ ./bin/singa-run.sh -conf examples/rbm/autoencoder.conf +</pre></div></div></div> +<div class="section"> +<h2><a name="Training_details"></a>Training details</h2> +<div class="section"> +<h3><a name="RBM1"></a>RBM1</h3> +<p><img src="../images/example-rbm1.png" align="center" width="200px" alt="" /> <span><b>Figure 1 - RBM1.</b></span></p> +<p>The neural net structure for training RBM1 is shown in Figure 1. The data layer and parser layer provides features for training RBM1. The visible layer (connected with parser layer) of RBM1 accepts the image feature (784 dimension). The hidden layer is set to have 1000 neurons (units). These two layers are configured as,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">layer{ + name: "RBMVis" + type: kRBMVis + srclayers:"mnist" + srclayers:"RBMHid" + rbm_conf{ + hdim: 1000 + } + param{ + name: "w1" + init{ + type: kGaussian + mean: 0.0 + std: 0.1 + } + } + param{ + name: "b11" + init{ + type: kConstant + value: 0.0 + } + } +} + +layer{ + name: "RBMHid" + type: kRBMHid + srclayers:"RBMVis" + rbm_conf{ + hdim: 1000 + } + param{ + name: "w1_" + share_from: "w1" + } + param{ + name: "b12" + init{ + type: kConstant + value: 0.0 + } + } +} +</pre></div></div> +<p>For RBM, the weight matrix is shared by the visible and hidden layers. For instance, <tt>w1</tt> is shared by <tt>vis</tt> and <tt>hid</tt> layers shown in Figure 1. In SINGA, we can configure the <tt>share_from</tt> field to enable <a href="param.html">parameter sharing</a> as shown above for the param <tt>w1</tt> and <tt>w1_</tt>.</p> +<p><a href="train-one-batch.html#contrastive-divergence">Contrastive Divergence</a> is configured as the algorithm for <a href="train-one-batch.html">TrainOneBatch</a>. Following Hinton’s paper, we configure the <a href="updater.html">updating protocol</a> as follows,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># Updater Configuration +updater{ + type: kSGD + momentum: 0.2 + weight_decay: 0.0002 + learning_rate{ + base_lr: 0.1 + type: kFixed + } +} +</pre></div></div> +<p>Since the parameters of RBM0 will be used to initialize the auto-encoder, we should configure the <tt>workspace</tt> field to specify a path for the checkpoint folder. For example, if we configure it as,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">cluster { + workspace: "examples/rbm/rbm1/" +} +</pre></div></div> +<p>Then SINGA will <a href="checkpoint.html">checkpoint the parameters</a> into <i>examples/rbm/rbm1/</i>.</p></div> +<div class="section"> +<h3><a name="RBM1"></a>RBM1</h3> +<p><img src="../images/example-rbm2.png" align="center" width="200px" alt="" /> <span><b>Figure 2 - RBM2.</b></span></p> +<p>Figure 2 shows the net structure of training RBM2. The visible units of RBM2 accept the output from the Sigmoid1 layer. The Inner1 layer is a <tt>InnerProductLayer</tt> whose parameters are set to the <tt>w1</tt> and <tt>b12</tt> learned from RBM1. The neural net configuration is (with layers for data layer and parser layer omitted).</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">layer{ + name: "Inner1" + type: kInnerProduct + srclayers:"mnist" + innerproduct_conf{ + num_output: 1000 + } + param{ name: "w1" } + param{ name: "b12"} +} + +layer{ + name: "Sigmoid1" + type: kSigmoid + srclayers:"Inner1" +} + +layer{ + name: "RBMVis" + type: kRBMVis + srclayers:"Sigmoid1" + srclayers:"RBMHid" + rbm_conf{ + hdim: 500 + } + param{ + name: "w2" + ... + } + param{ + name: "b21" + ... + } +} + +layer{ + name: "RBMHid" + type: kRBMHid + srclayers:"RBMVis" + rbm_conf{ + hdim: 500 + } + param{ + name: "w2_" + share_from: "w2" + } + param{ + name: "b22" + ... + } +} +</pre></div></div> +<p>To load w0 and b02 from RBM0’s checkpoint file, we configure the <tt>checkpoint_path</tt> as,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">checkpoint_path: "examples/rbm/rbm1/checkpoint/step6000-worker0" +cluster{ + workspace: "examples/rbm/rbm2" +} +</pre></div></div> +<p>The workspace is changed for checkpointing <tt>w2</tt>, <tt>b21</tt> and <tt>b22</tt> into <i>examples/rbm/rbm2/</i>.</p></div> +<div class="section"> +<h3><a name="RBM3"></a>RBM3</h3> +<p><img src="../images/example-rbm3.png" align="center" width="200px" alt="" /> <span><b>Figure 3 - RBM3.</b></span></p> +<p>Figure 3 shows the net structure of training RBM3. In this model, a layer with 250 units is added as the hidden layer of RBM3. The visible units of RBM3 accepts output from Sigmoid2 layer. Parameters of Inner1 and Innner2 are set to <tt>w1,b12,w2,b22</tt> which can be load from the checkpoint file of RBM2, i.e., “examples/rbm/rbm2/”.</p></div> +<div class="section"> +<h3><a name="RBM4"></a>RBM4</h3> +<p><img src="../images/example-rbm4.png" align="center" width="200px" alt="" /> <span><b>Figure 4 - RBM4.</b></span></p> +<p>Figure 4 shows the net structure of training RBM4. It is similar to Figure 3, but according to <a class="externalLink" href="http://www.cs.toronto.edu/~hinton/science.pdf";>Hinton’s science paper</a>, the hidden units of the top RBM (RBM4) have stochastic real-valued states drawn from a unit variance Gaussian whose mean is determined by the input from the RBM’s logistic visible units. So we add a <tt>gaussian</tt> field in the RBMHid layer to control the sampling distribution (Gaussian or Bernoulli). In addition, this RBM has a much smaller learning rate (0.001). The neural net configuration for the RBM4 and the updating protocol is (with layers for data layer and parser layer omitted),</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># Updater Configuration +updater{ + type: kSGD + momentum: 0.9 + weight_decay: 0.0002 + learning_rate{ + base_lr: 0.001 + type: kFixed + } +} + +layer{ + name: "RBMVis" + type: kRBMVis + srclayers:"Sigmoid3" + srclayers:"RBMHid" + rbm_conf{ + hdim: 30 + } + param{ + name: "w4" + ... + } + param{ + name: "b41" + ... + } +} + +layer{ + name: "RBMHid" + type: kRBMHid + srclayers:"RBMVis" + rbm_conf{ + hdim: 30 + gaussian: true + } + param{ + name: "w4_" + share_from: "w4" + } + param{ + name: "b42" + ... + } +} +</pre></div></div></div> +<div class="section"> +<h3><a name="Auto-encoder"></a>Auto-encoder</h3> +<p>In the fine-tuning stage, the 4 RBMs are “unfolded” to form encoder and decoder networks that are initialized using the parameters from the previous 4 RBMs.</p> +<p><img src="../images/example-autoencoder.png" align="center" width="500px" alt="" /> <span><b>Figure 5 - Auto-Encoders.</b></span></p> +<p>Figure 5 shows the neural net structure for training the auto-encoder. <a href="train-one-batch.html">Back propagation (kBP)</a> is configured as the algorithm for <tt>TrainOneBatch</tt>. We use the same cluster configuration as RBM models. For updater, we use <a href="updater.html#adagradupdater">AdaGrad</a> algorithm with fixed learning rate.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">### Updater Configuration +updater{ + type: kAdaGrad + learning_rate{ + base_lr: 0.01 + type: kFixed + } +} +</pre></div></div> +<p>According to <a class="externalLink" href="http://www.cs.toronto.edu/~hinton/science.pdf";>Hinton’s science paper</a>, we configure a EuclideanLoss layer to compute the reconstruction error. The neural net configuration is (with some of the middle layers omitted),</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">layer{ name: "data" } +layer{ name:"mnist" } +layer{ + name: "Inner1" + param{ name: "w1" } + param{ name: "b12" } +} +layer{ name: "Sigmoid1" } +... +layer{ + name: "Inner8" + innerproduct_conf{ + num_output: 784 + transpose: true + } + param{ + name: "w8" + share_from: "w1" + } + param{ name: "b11" } +} +layer{ name: "Sigmoid8" } + +# Euclidean Loss Layer Configuration +layer{ + name: "loss" + type:kEuclideanLoss + srclayers:"Sigmoid8" + srclayers:"mnist" +} +</pre></div></div> +<p>To load pre-trained parameters from the 4 RBMs’ checkpoint file we configure <tt>checkpoint_path</tt> as</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">### Checkpoint Configuration +checkpoint_path: "examples/rbm/checkpoint/rbm1/checkpoint/step6000-worker0" +checkpoint_path: "examples/rbm/checkpoint/rbm2/checkpoint/step6000-worker0" +checkpoint_path: "examples/rbm/checkpoint/rbm3/checkpoint/step6000-worker0" +checkpoint_path: "examples/rbm/checkpoint/rbm4/checkpoint/step6000-worker0" +</pre></div></div></div></div> +<div class="section"> +<h2><a name="Visualization_Results"></a>Visualization Results</h2> + +<div> +<img src="../images/rbm-weight.PNG" align="center" width="300px" alt="" /> + +<img src="../images/rbm-feature.PNG" align="center" width="300px" alt="" /> +<br /> +<span><b>Figure 6 - Bottom RBM weight matrix.</b></span> +  +  +  +  + +<span><b>Figure 7 - Top layer features.</b></span> +</div> +<p>Figure 6 visualizes sample columns of the weight matrix of RBM1, We can see the Gabor-like filters are learned. Figure 7 depicts the features extracted from the top-layer of the auto-encoder, wherein one point represents one image. Different colors represent different digits. We can see that most images are well clustered according to the ground truth.</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html>
![https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"](https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"){kind=link}
![http://incubator.apache.org/images/egg-logo.png"](http://incubator.apache.org/images/egg-logo.png"){kind=link}