Author: buildbot
Date: Wed Sep 2 08:15:15 2015
New Revision: 964001
Log:
Staging update by buildbot for singa
Added:
websites/staging/singa/trunk/content/docs/data.html
Modified:
websites/staging/singa/trunk/content/ (props changed)
websites/staging/singa/trunk/content/community.html
websites/staging/singa/trunk/content/community/issue-tracking.html
websites/staging/singa/trunk/content/community/mail-lists.html
websites/staging/singa/trunk/content/community/source-repository.html
websites/staging/singa/trunk/content/community/team-list.html
websites/staging/singa/trunk/content/develop/contribute-code.html
websites/staging/singa/trunk/content/develop/contribute-docs.html
websites/staging/singa/trunk/content/develop/how-contribute.html
websites/staging/singa/trunk/content/develop/schedule.html
websites/staging/singa/trunk/content/docs.html
websites/staging/singa/trunk/content/docs/architecture.html
websites/staging/singa/trunk/content/docs/checkpoint.html
websites/staging/singa/trunk/content/docs/cnn.html
websites/staging/singa/trunk/content/docs/code-structure.html
websites/staging/singa/trunk/content/docs/communication.html
Propchange: websites/staging/singa/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Wed Sep 2 08:15:15 2015
@@ -1 +1 @@
-1696297
+1700726
Modified: websites/staging/singa/trunk/content/community.html
==============================================================================
--- websites/staging/singa/trunk/content/community.html (original)
+++ websites/staging/singa/trunk/content/community.html Wed Sep 2 08:15:15 2015
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache SINGA – Community</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" />
Modified: websites/staging/singa/trunk/content/community/issue-tracking.html
==============================================================================
--- websites/staging/singa/trunk/content/community/issue-tracking.html
(original)
+++ websites/staging/singa/trunk/content/community/issue-tracking.html Wed Sep
2 08:15:15 2015
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache SINGA – Issue Tracking</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
Modified: websites/staging/singa/trunk/content/community/mail-lists.html
==============================================================================
--- websites/staging/singa/trunk/content/community/mail-lists.html (original)
+++ websites/staging/singa/trunk/content/community/mail-lists.html Wed Sep 2
08:15:15 2015
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache SINGA – Project Mailing Lists</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
Modified: websites/staging/singa/trunk/content/community/source-repository.html
==============================================================================
--- websites/staging/singa/trunk/content/community/source-repository.html
(original)
+++ websites/staging/singa/trunk/content/community/source-repository.html Wed
Sep 2 08:15:15 2015
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache SINGA – Source Repository</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
Modified: websites/staging/singa/trunk/content/community/team-list.html
==============================================================================
--- websites/staging/singa/trunk/content/community/team-list.html (original)
+++ websites/staging/singa/trunk/content/community/team-list.html Wed Sep 2
08:15:15 2015
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache SINGA – The SINGA Team</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
Modified: websites/staging/singa/trunk/content/develop/contribute-code.html
==============================================================================
--- websites/staging/singa/trunk/content/develop/contribute-code.html (original)
+++ websites/staging/singa/trunk/content/develop/contribute-code.html Wed Sep
2 08:15:15 2015
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache SINGA – How to Contribute Code</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
Modified: websites/staging/singa/trunk/content/develop/contribute-docs.html
==============================================================================
--- websites/staging/singa/trunk/content/develop/contribute-docs.html (original)
+++ websites/staging/singa/trunk/content/develop/contribute-docs.html Wed Sep
2 08:15:15 2015
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache SINGA – How to Contribute Documentation</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
Modified: websites/staging/singa/trunk/content/develop/how-contribute.html
==============================================================================
--- websites/staging/singa/trunk/content/develop/how-contribute.html (original)
+++ websites/staging/singa/trunk/content/develop/how-contribute.html Wed Sep 2
08:15:15 2015
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache SINGA – How to Contribute to SINGA</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
Modified: websites/staging/singa/trunk/content/develop/schedule.html
==============================================================================
--- websites/staging/singa/trunk/content/develop/schedule.html (original)
+++ websites/staging/singa/trunk/content/develop/schedule.html Wed Sep 2
08:15:15 2015
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache SINGA – Development Schedule</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
Modified: websites/staging/singa/trunk/content/docs.html
==============================================================================
--- websites/staging/singa/trunk/content/docs.html (original)
+++ websites/staging/singa/trunk/content/docs.html Wed Sep 2 08:15:15 2015
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache SINGA – Documentation</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" />
Modified: websites/staging/singa/trunk/content/docs/architecture.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/architecture.html (original)
+++ websites/staging/singa/trunk/content/docs/architecture.html Wed Sep 2
08:15:15 2015
@@ -1,15 +1,15 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
- <title>Apache SINGA – System Architecture</title>
+ <title>Apache SINGA – </title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
<link rel="stylesheet" href="../css/site.css" />
<link rel="stylesheet" href="../css/print.css" media="print" />
@@ -189,7 +189,7 @@
Apache SINGA</a>
<span class="divider">/</span>
</li>
- <li class="active ">System Architecture</li>
+ <li class="active "></li>
@@ -423,14 +423,15 @@
<div id="bodyColumn" class="span10" >
- <div class="section">
-<h2><a name="System_Architecture"></a>System Architecture</h2>
-<hr />
+ <p>— layout: post title: Architecture category : docs</p>
+<div class="section">
+<h2><a name="tags_:_architecture"></a>tags : [architecture]</h2>
+<p>{% include JB/setup %}</p>
<div class="section">
<h3><a name="Logical_Architecture"></a>Logical Architecture</h3>
-<p><img src="../images/distributed/logical.png" style="width: 550px" alt="" />
+<p><img src="http://singa.incubator.apache.org/assets/image/logical.png";
style="width: 550px" alt="" />
<p><b> Fig.1 - Logical system architecture</b></p>
-<p>SINGA has flexible architecture to support different distributed <a
href="frameworks.html">training frameworks</a> (both synchronous and
asynchronous). The logical system architecture is shown in Fig.1. The
architecture consists of multiple server groups and worker groups:</p>
+<p>SINGA has flexible architecture to support different distributed <a
class="externalLink"
href="http://singa.incubator.apache.org/docs/frameworks.html";>training
frameworks</a> (both synchronous and asynchronous). The logical system
architecture is shown in Fig.1. The architecture consists of multiple server
groups and worker groups:</p>
<ul>
@@ -438,7 +439,7 @@
<li><b>Worker group</b> Each worker group communicates with only one server
group. A worker group trains a complete model replica against a partition of
the training dataset, and is responsible for computing parameter gradients.
All worker groups run and communicate with the corresponding server groups
asynchronously. However, inside each worker group, the workers synchronously
compute parameter updates for the model replica.</li>
</ul>
-<p>There are different strategies to distribute the training workload among
workers within a group: </p>
+<p>There are different strategies to distribute the training workload among
workers within a group:</p>
<ul>
@@ -450,7 +451,7 @@
</ul></div>
<div class="section">
<h3><a name="Implementation"></a>Implementation</h3>
-<p>In SINGA, servers and workers are execution units running in separate
threads. They communicate through <a href="communication.html">messages</a>.
Every process runs the main thread as a stub that aggregates local messages and
forwards them to corresponding (remote) receivers.</p>
+<p>In SINGA, servers and workers are execution units running in separate
threads. They communicate through <a class="externalLink"
href="http://singa.incubator.apache.org/docs/communication.html";>messages</a>.
Every process runs the main thread as a stub that aggregates local messages and
forwards them to corresponding (remote) receivers.</p>
<p>Each server group and worker group have a <i>ParamShard</i> object
representing a complete model replica. If workers and servers resident in the
same process, their <i>ParamShard</i> (partitions) can be configured to share
the same memory space. In this case, the messages transferred between different
execution units just contain pointers to the data, which reduces the
communication cost. Unlike in inter-process cases, the messages have to include
the parameter values.</p></div></div>
</div>
</div>
Modified: websites/staging/singa/trunk/content/docs/checkpoint.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/checkpoint.html (original)
+++ websites/staging/singa/trunk/content/docs/checkpoint.html Wed Sep 2
08:15:15 2015
@@ -1,15 +1,15 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
- <title>Apache SINGA – Checkpoint and Resume</title>
+ <title>Apache SINGA – </title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
<link rel="stylesheet" href="../css/site.css" />
<link rel="stylesheet" href="../css/print.css" media="print" />
@@ -189,7 +189,7 @@
Apache SINGA</a>
<span class="divider">/</span>
</li>
- <li class="active ">Checkpoint and Resume</li>
+ <li class="active "></li>
@@ -423,80 +423,84 @@
<div id="bodyColumn" class="span10" >
- <div class="section">
-<h2><a name="Checkpoint_and_Resume"></a>Checkpoint and Resume</h2>
-<hr />
+ <p>— layout: post title: Checkpoint and Resume category :
docs</p>
<div class="section">
-<h3><a name="Applications_of_checkpoint"></a>Applications of checkpoint</h3>
-<p>By taking checkpoints of model parameters, we can</p>
+<h2><a name="tags_:_checkpoint_restore"></a>tags : [checkpoint, restore]</h2>
+<p>{% include JB/setup %}</p>
+<p>SINGA checkpoints model parameters onto disk periodically according to user
configured frequency. By checkpointing model parameters, we can</p>
<ol style="list-style-type: decimal">
<li>
-<p>Restore (resume) the training from the last checkpoint. For example, if the
program crashes before finishing all training steps.</p></li>
+<p>resume the training from the last checkpointing. For example, if the
program crashes before finishing all training steps, we can continue the
training using checkpoint files.</p></li>
<li>
-<p>Use them as pre-training results for a similar model. For example, the
parameters from training a RBM model can be used to initialize a <a
href="auto-encoder.html">deep auto-encoder</a> model.</p></li>
+<p>use them to initialize a similar model. For example, the parameters from
training a RBM model can be used to initialize a <a class="externalLink"
href="http://singa.incubator.apache.org/docs/rbm";>deep auto-encoder</a>
model.</p></li>
</ol></div>
<div class="section">
-<h3><a name="Instructions_for_checkpoint_and_resume"></a>Instructions for
checkpoint and resume</h3>
-<p>Checkpoint is controlled by two model configuration fields:
<tt>checkpoint_after</tt> (start checkpoint after this number of training
steps) and <tt>checkpoint_frequency</tt>. The checkpoint files are located at
<tt>WORKSPACE/checkpoint/stepSTEP-workerWORKERID.bin</tt>.</p>
-<p>The following configuration shows an example,</p>
+<h2><a name="Configuration"></a>Configuration</h2>
+<p>Checkpointing is controlled by two configuration fields:</p>
+
+<ul>
+
+<li><tt>checkpoint_after</tt>, start checkpointing after this number of
training steps,</li>
+
+<li><tt>checkpoint_freq</tt>, frequency of doing checkpointing.</li>
+</ul>
+<p>For example,</p>
<div class="source">
-<div class="source"><pre class="prettyprint">model {
- ...
- checkpoint_after: 100
- checkpoint_frequency: 300
- ...
-}
+<div class="source"><pre class="prettyprint"># job.conf
+workspace: "WORKSPACE"
+checkpoint_after: 100
+checkpoint_frequency: 300
+...
</pre></div></div>
-<p>After training for 700 steps, under WORKSPACE/checkpoint folder, there
would be two checkpoint files (training on single node):</p>
+<p>Checkpointing files are located at
<i>WORKSPACE/checkpoint/stepSTEP-workerWORKERID.bin</i>. For the above
configuration, after training for 700 steps, there would be two checkpointing
files,</p>
<div class="source">
<div class="source"><pre class="prettyprint">step400-worker0.bin
step700-worker0.bin
-</pre></div></div>
+</pre></div></div></div>
<div class="section">
-<h4><a name="Application_1"></a>Application 1</h4>
-<p>We can resume the training from the last checkpoint (i.e., step 700) by:</p>
+<h2><a name="Application_-_resuming_training"></a>Application - resuming
training</h2>
+<p>We can resume the training from the last checkpoint (i.e., step 700) by,</p>
<div class="source">
-<div class="source"><pre class="prettyprint">./bin/singa-run.sh
-workspace=WORKSPACE --resume
-</pre></div></div></div>
+<div class="source"><pre class="prettyprint">./bin/singa-run.sh -conf JOB_CONF
-resume
+</pre></div></div>
+<p>There is no change to the job configuration.</p></div>
<div class="section">
-<h4><a name="Application_2"></a>Application 2</h4>
-<p>We can also use the checkpoint file from step 400 as the pre-trained model
for a new model by configuring the job.conf of the new model as:</p>
+<h2><a name="Application_-_model_initialization"></a>Application - model
initialization</h2>
+<p>We can also use the checkpointing file from step 400 to initialize a new
model by configuring the new job as,</p>
<div class="source">
-<div class="source"><pre class="prettyprint">model {
- ...
- checkpoint : "WORKSPACE/checkpoint/step400-worker0.bin"
- ...
-}
+<div class="source"><pre class="prettyprint"># job.conf
+checkpoint : "WORKSPACE/checkpoint/step400-worker0.bin"
+...
</pre></div></div>
-<p>If there are multiple checkpoint files for the same snapshot due to model
partitioning, all the checkpoint files should be added:</p>
+<p>If there are multiple checkpointing files for the same snapshot due to
model partitioning, all the checkpointing files should be added,</p>
<div class="source">
-<div class="source"><pre class="prettyprint">model {
- ...
- checkpoint : "WORKSPACE/checkpoint/step400-worker0.bin"
- checkpoint : "WORKSPACE/checkpoint/step400-worker1.bin"
- ...
-}
+<div class="source"><pre class="prettyprint"># job.conf
+checkpoint : "WORKSPACE/checkpoint/step400-worker0.bin"
+checkpoint : "WORKSPACE/checkpoint/step400-worker1.bin"
+...
</pre></div></div>
-<p>The launching command is the same as starting a new job</p>
+<p>The training command is the same as starting a new job,</p>
<div class="source">
-<div class="source"><pre class="prettyprint">./bin/singa-run.sh
-workspace=WORKSPACE
-</pre></div></div></div></div>
+<div class="source"><pre class="prettyprint">./bin/singa-run.sh -conf JOB_CONF
+</pre></div></div>
+<p>{% comment %}</p></div>
<div class="section">
-<h3><a name="Implementation_details"></a>Implementation details</h3>
-<p>The checkpoint is done in the Worker class and controlled by two model
configuration fields: <tt>checkpoint_after</tt> and
<tt>checkpoint_frequency</tt>. Only Params owning the param values from the
first group are dumped onto into checkpoint files. For one Param object, its
name, version and values are saved. It is possible that the snapshot is
separated into multiple files because the neural net is partitioned into
multiple workers.</p>
-<p>The Worker’s InitLocalParam will initialize Params from checkpoint
files if the <tt>checkpoint</tt> field is set. Otherwise it randomly initialize
them using user configured initialization method. The Param objects are matched
based on name. If the Param is not configured with a name, NeuralNet class will
automatically create one for it based on the name of the layer to which the
Param object belongs. The <tt>checkpoint</tt> can be set by users (Application
1) or by the Resume function (Application 2) of the Trainer class, which finds
the files for the latest snapshot and add them to the <tt>checkpoint</tt>
filed. It also sets the <tt>step</tt> field of model configuration to the
checkpoint step (extracted from file name).</p></div>
+<h2><a name="Advanced_user_guide"></a>Advanced user guide</h2>
+<p>Checkpointing is done in the <a class="externalLink"
href="http://singa.incubator.apache.org/api/classsinga_1_1Worker.html";>Worker
class</a>. Only <tt>Param</tt>s from the first group are dumped into
checkpointing files. For a <tt>Param</tt> object, its name, version and values
are saved. It is possible that the snapshot is separated into multiple files
because the neural net is partitioned into multiple workers.</p>
+<p>The Worker’s <tt>InitLocalParam</tt> function will initialize
parameters from checkpointing files if the <tt>checkpoint</tt> field is set.
Otherwise it randomly initialize them using user configured initialization
method. The <tt>Param</tt> objects are matched based on name. If a
<tt>Param</tt> object is not configured with a name, <tt>NeuralNet</tt> class
will automatically create one for it based on the name of the layer. The
<tt>checkpoint</tt> can be set by users (Application 1) or by the
<tt>Resume</tt> function (Application 2) of the Trainer class, which finds the
files for the latest snapshot and add them to the <tt>checkpoint</tt> filed. It
also sets the <tt>step</tt> field of model configuration to the checkpoint step
(extracted from file name).</p>
<div class="section">
<h3><a name="Caution"></a>Caution</h3>
-<p>Both two applications must be taken carefully when Param objects are
partitioned due to model partitioning. Because if the training is done using 2
workers, while the new model (or continue training) is trained with 3 workers,
then the same original Param object is partitioned in different ways and hence
cannot be matched.</p></div></div>
+<p>Both two applications must be taken carefully when <tt>Param</tt> objects
are partitioned due to model partitioning. Because if the training is done
using 2 workers, while the new model (or continue training) is trained with 3
workers, then the same original <tt>Param</tt> object is partitioned in
different ways and hence cannot be matched.</p>
+<p>{% endcomment %}</p></div></div>
</div>
</div>
</div>
Modified: websites/staging/singa/trunk/content/docs/cnn.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/cnn.html (original)
+++ websites/staging/singa/trunk/content/docs/cnn.html Wed Sep 2 08:15:15 2015
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache SINGA – </title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
@@ -21,7 +21,7 @@
<script type="text/javascript"
src="../js/apache-maven-fluido-1.4.min.js"></script>
- <meta name="Notice" content="Licensed to the Apache Software Foundation
(ASF) under one or more contributor license agreements. See the
NOTICE file distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at .
http://www.apache.org/licenses/LICENSE-2.0 . Unless
required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS"
BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
or implied. See the License for the specific language governing
permissions and limitations under the License." /> </hea
d>
+ </head>
<body class="topBarEnabled">
@@ -425,62 +425,258 @@
<div id="bodyColumn" class="span10" >
-
-<p>This example will show you how to use SINGA to train a CNN model using
cifar10 dataset.</p>
+ <p>— layout: post title: Example — Convolution
Neural Network category : docs</p>
<div class="section">
+<h2><a name="tags_:_cnn_example"></a>tags : [cnn, example]</h2>
+<p>{% include JB/setup %}</p>
+<p>Convolutional neural network (CNN) is a type of feed-forward artificial
neural network widely used for image and video classification. In this example,
we will use a deep CNN model to do image classification for the <a
class="externalLink" href="http://www.cs.toronto.edu/~kriz/cifar.html";>CIFAR10
dataset</a>.</p></div>
<div class="section">
-<h3><a name="Prepare_for_the_data"></a>Prepare for the data</h3>
+<h2><a name="Running_instructions"></a>Running instructions</h2>
+<p>Please refer to the <a class="externalLink"
href="http://singa.incubator.apache.org/docs/installation";>installation</a>
page for instructions on building SINGA, and the <a class="externalLink"
href="http://singa.incubator.apache.org/docs/quick-start";>quick start</a> for
instructions on starting zookeeper.</p>
+<p>We have provided scripts for preparing the training and test dataset in
<i>examples/cifar10/</i>.</p>
-<ul>
-
-<li>First go to the <tt>example/cifar10/</tt> folder for preparing the
dataset. There should be a makefile example called Makefile.example in the
folder. Run the command <tt>cp Makefile.example Makefile</tt> to generate the
makefile. Then run the command <tt>make download</tt> and <tt>make create</tt>
in the current folder to download cifar10 dataset and prepare for the training
and testing datashard.</li>
-</ul></div>
+<div class="source">
+<div class="source"><pre class="prettyprint"># in examples/cifar10
+$ cp Makefile.example Makefile
+$ make download
+$ make create
+</pre></div></div>
+<p>After the datasets are prepared, we start the training by</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">./bin/singa-run.sh -conf
examples/cifar10/job.conf
+</pre></div></div>
+<p>After it is started, you should see output like</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">Record job information to
/tmp/singa-log/job-info/job-2-20150817-055601
+Executing : ./singa -conf /xxx/incubator-singa/examples/cifar10/job.conf
-singa_conf /xxx/incubator-singa/conf/singa.conf -singa_job 2
+E0817 06:56:18.868259 33849 cluster.cc:51] proc #0 -> 192.168.5.128:49152
(pid = 33849)
+E0817 06:56:18.928452 33871 server.cc:36] Server (group = 0, id = 0) start
+E0817 06:56:18.928469 33872 worker.cc:134] Worker (group = 0, id = 0) start
+E0817 06:57:13.657302 33849 trainer.cc:373] Test step-0, loss : 2.302588,
accuracy : 0.077900
+E0817 06:57:17.626708 33849 trainer.cc:373] Train step-0, loss : 2.302578,
accuracy : 0.062500
+E0817 06:57:24.142645 33849 trainer.cc:373] Train step-30, loss : 2.302404,
accuracy : 0.131250
+E0817 06:57:30.813354 33849 trainer.cc:373] Train step-60, loss : 2.302248,
accuracy : 0.156250
+E0817 06:57:37.556655 33849 trainer.cc:373] Train step-90, loss : 2.301849,
accuracy : 0.175000
+E0817 06:57:44.971276 33849 trainer.cc:373] Train step-120, loss : 2.301077,
accuracy : 0.137500
+E0817 06:57:51.801949 33849 trainer.cc:373] Train step-150, loss : 2.300410,
accuracy : 0.135417
+E0817 06:57:58.682281 33849 trainer.cc:373] Train step-180, loss : 2.300067,
accuracy : 0.127083
+E0817 06:58:05.578366 33849 trainer.cc:373] Train step-210, loss : 2.300143,
accuracy : 0.154167
+E0817 06:58:12.518497 33849 trainer.cc:373] Train step-240, loss : 2.295912,
accuracy : 0.185417
+</pre></div></div>
+<p>After the training of some steps (depends on the setting) or the job is
finished, SINGA will <a class="externalLink"
href="http://singa.incubator.apache.org/docs/checkpoint";>checkpoint</a> the
model parameters.</p></div>
+<div class="section">
+<h2><a name="Details"></a>Details</h2>
+<p>To train a model in SINGA, you need to prepare the datasets, and a job
configuration which specifies the neural net structure, training algorithm (BP
or CD), SGD update algorithm (e.g. Adagrad), number of training/test steps,
etc.</p>
+<div class="section">
+<h3><a name="Data_preparation"></a>Data preparation</h3>
+<p>Before using SINGA, you need to write a program to pre-process the dataset
you use to a format that SINGA can read. Please refer to the <a
class="externalLink"
href="http://singa.incubator.apache.org/docs/data#example---cifar-dataset";>Data
Preparation</a> to get details about preparing this CIFAR10 dataset.</p></div>
<div class="section">
-<h3><a name="Set_job_configuration."></a>Set job configuration.</h3>
+<h3><a name="Neural_net"></a>Neural net</h3>
+<p>Figure 1 shows the net structure of the CNN model we used in this example,
which is set following <a class="externalLink"
href="https://code.google.com/p/cuda-convnet/source/browse/trunk/example-layers/layers-18pct.cfg.";>this
page</a> The dashed circle represents one feature transformation stage, which
generally has four layers as shown in the figure. Sometimes the rectifier layer
and normalization layer is omitted or swapped in one stage. For this example,
there are 3 such stages.</p>
+<p>Next we follow the guide in <a class="externalLink"
href="http://singa.incubator.apache.org/docs/neural-net";>neural net page</a>
and <a class="externalLink"
href="http://singa.incubator.apache.org/docs/layer";>layer page</a> to write the
neural net configuration.</p>
+
+<div style="text-align: center">
+<img src="http://singa.incubator.apache.org/assets/image/cnn-example.png";
style="width: 200px" alt="" /> <br />
+<b>Figure 1 - Net structure of the CNN example.</b></img>
+</div>
<ul>
-<li>If you just want to use the training model provided in this example, you
can just use job.conf file in current directory. Fig. 1 gives an example of CNN
struture. In this example, we define a CNN model that contains 3
convolution+relu+maxpooling+normalization layers. If you want to learn more
about how it is configured, you can go to <a class="externalLink"
href="http://singa.incubator.apache.org/docs/model-config.html";>Model
Configuration</a> to get details.</li>
+<li>
+<p>We configure a <a class="externalLink"
href="http://singa.incubator.apache.org/docs/layer#data-layers";>data layer</a>
to read the training/testing <tt>Records</tt> from <tt>DataShard</tt>.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">layer{
+ name: "data"
+ type: kShardData
+ sharddata_conf {
+ path: "examples/cifar10/cifar10_train_shard"
+ batchsize: 16
+ random_skip: 5000
+ }
+ exclude: kTest # exclude this layer for the testing net
+ }
+layer{
+ name: "data"
+ type: kShardData
+ sharddata_conf {
+ path: "examples/cifar10/cifar10_test_shard"
+ batchsize: 100
+ }
+ exclude: kTrain # exclude this layer for the training net
+ }
+</pre></div></div></li>
+
+<li>
+<p>We configure two <a class="externalLink"
href="http://singa.incubator.apache.org/docs/layer#parser-layers";>parser
layers</a> to extract the image feature and label from <tt>Records</tt>s loaded
by the <i>data</i> layer.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">layer{
+ name:"rgb"
+ type: kRGBImage
+ srclayers: "data"
+ rgbimage_conf {
+ meanfile: "examples/cifar10/image_mean.bin" # normalize image
feature
+ }
+ }
+layer{
+ name: "label"
+ type: kLabel
+ srclayers: "data"
+ }
+</pre></div></div></li>
</ul>
-<div style="text-align: center">
-<img src="../images/dcnn-cifar10.png" style="width: 280px" alt="" /> <br
/>Fig. 1: CNN example </img>
-</div></div>
-<div class="section">
-<h3><a name="Run_SINGA"></a>Run SINGA</h3>
-
<ul>
-<li>All script of SINGA should be run in the root folder of SINGA. First you
need to start the zookeeper service if zookeeper is not started. The command is
<tt>./bin/zk-service start</tt>. Then you can run the command
<tt>./bin/singa-run.sh -conf examples/cifar10/job.conf</tt> to start a SINGA
job using examples/cifar10/job.conf as the job configuration. After it is
started, you should get a screenshots like the following:</li>
+<li>
+<p>We configure layers for the feature transformation as follows (all layers
are built-in layers in SINGA; hyper-parameters of these layers are set
according to <a class="externalLink"
href="https://code.google.com/p/cuda-convnet/source/browse/trunk/example-layers/layers-18pct.cfg";>Alex’s
setting</a>).</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">layer {
+ name: "conv1"
+ type: kConvolution
+ srclayers: "rgb"
+ convolution_conf {
+ num_filters: 32
+ kernel: 5
+ stride: 1
+ pad:2
+ }
+ param {
+ name: "w1"
+ init {
+ type:kGaussian
+ std:0.0001
+ }
+ }
+ param {
+ name: "b1"
+ lr_scale:2.0
+ init {
+ type: kConstant
+ value:0
+ }
+ }
+ }
+
+ layer {
+ name: "pool1"
+ type: kPooling
+ srclayers: "conv1"
+ pooling_conf {
+ pool: MAX
+ kernel: 3
+ stride: 2
+ }
+ }
+ layer {
+ name: "relu1"
+ type: kReLU
+ srclayers:"pool1"
+ }
+ layer {
+ name: "norm1"
+ type: kLRN
+ lrn_conf {
+ local_size: 3
+ alpha: 5e-05
+ beta: 0.75
+ }
+ srclayers:"relu1"
+ }
+</pre></div></div></li>
</ul>
+<p>The configurations for another 2 stages are omitted here.</p>
+<ul>
+
+<li>
+<p>There is a <a class="externalLink"
href="http://singa.incubator.apache.org/docs/layer#innerproductlayer";>inner
product layer</a> after the 3 transformation stages, which is configured with
10 output units, i.e., the number of total labels. The weight matrix param is
configured with a large weight decay scale to reduce the over-fitting.</p>
+
<div class="source">
-<div class="source"><pre class="prettyprint"> xxx@yyy:zzz/incubator-singa$
./bin/singa-run.sh -conf examples/cifar10/job.conf
- Unique JOB_ID is 2
- Record job information to /tmp/singa-log/job-info/job-2-20150817-055601
- Executing : ./singa -conf /xxx/incubator-singa/examples/cifar10/job.conf
-singa_conf /xxx/incubator-singa/conf/singa.conf -singa_job 2
- E0817 06:56:18.868259 33849 cluster.cc:51] proc #0 ->
192.168.5.128:49152 (pid = 33849)
- E0817 06:56:18.928452 33871 server.cc:36] Server (group = 0, id = 0) start
- E0817 06:56:18.928469 33872 worker.cc:134] Worker (group = 0, id = 0) start
- E0817 06:57:13.657302 33849 trainer.cc:373] Test step-0, loss : 2.302588,
accuracy : 0.077900
- E0817 06:57:17.626708 33849 trainer.cc:373] Train step-0, loss : 2.302578,
accuracy : 0.062500
- E0817 06:57:24.142645 33849 trainer.cc:373] Train step-30, loss :
2.302404, accuracy : 0.131250
- E0817 06:57:30.813354 33849 trainer.cc:373] Train step-60, loss :
2.302248, accuracy : 0.156250
- E0817 06:57:37.556655 33849 trainer.cc:373] Train step-90, loss :
2.301849, accuracy : 0.175000
- E0817 06:57:44.971276 33849 trainer.cc:373] Train step-120, loss :
2.301077, accuracy : 0.137500
- E0817 06:57:51.801949 33849 trainer.cc:373] Train step-150, loss :
2.300410, accuracy : 0.135417
- E0817 06:57:58.682281 33849 trainer.cc:373] Train step-180, loss :
2.300067, accuracy : 0.127083
- E0817 06:58:05.578366 33849 trainer.cc:373] Train step-210, loss :
2.300143, accuracy : 0.154167
- E0817 06:58:12.518497 33849 trainer.cc:373] Train step-240, loss :
2.295912, accuracy : 0.185417
-</pre></div></div>
-<p>After the training of some steps (depends on the setting) or the job is
finished, SINGA will checkpoint the current parameter. In the next time, you
can train (or use for your application) by loading the checkpoint. Please refer
to <a class="externalLink"
href="http://singa.incubator.apache.org/docs/checkpoint.html";>Checkpoint</a>
for the use of checkpoint.</p></div>
+<div class="source"><pre class="prettyprint">layer {
+ name: "ip1"
+ type: kInnerProduct
+ srclayers:"pool3"
+ innerproduct_conf {
+ num_output: 10
+ }
+ param {
+ name: "w4"
+ wd_scale:250
+ init {
+ type:kGaussian
+ std:0.01
+ }
+ }
+ param {
+ name: "b4"
+ lr_scale:2.0
+ wd_scale:0
+ init {
+ type: kConstant
+ value:0
+ }
+ }
+ }
+</pre></div></div></li>
+
+<li>
+<p>The last layer is a <a class="externalLink"
href="http://singa.incubator.apache.org/docs/layer#softmaxloss";>Softmax loss
layer</a></p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"> layer{
+ name: "loss"
+ type: kSoftmaxLoss
+ softmaxloss_conf{
+ topk:1
+ }
+ srclayers:"ip1"
+ srclayers: "label"
+ }
+</pre></div></div></li>
+</ul></div>
<div class="section">
-<h3><a name="Build_your_own_model"></a>Build your own model</h3>
+<h3><a name="Updater"></a>Updater</h3>
+<p>The <a class="externalLink"
href="http://singa.incubator.apache.org/docs/updater#updater";>normal SGD
updater</a> is selected. The learning rate is changed like stairs, and is
configured using the <a class="externalLink"
href="http://singa.incubator.apache.org/docs/updater#kfixedstep";>kFixedStep</a>
type.</p>
-<ul>
-
-<li>If you want to specify you own model, then you need to decribe it in the
job.conf file. It should contain the neurualnet structure, training
algorithm(backforward or contrastive divergence etc.), SGD update
algorithm(e.g. Adagrad), number of training/test steps and training/test
frequency, and display features and etc. SINGA will read job.conf as a Google
protobuf class <a href="../src/proto/job.proto">JobProto</a>. You can also
refer to the <a class="externalLink"
href="http://singa.incubator.apache.org/docs/programmer-guide.html";>Programmer
Guide</a> to get details.</li>
-</ul></div></div>
+<div class="source">
+<div class="source"><pre class="prettyprint">updater{
+ type: kSGD
+ weight_decay:0.004
+ learning_rate {
+ type: kFixedStep
+ fixedstep_conf:{
+ step:0 # lr for step 0-60000 is 0.001
+ step:60000 # lr for step 60000-65000 is 0.0001
+ step:65000 # lr for step 650000- is 0.00001
+ step_lr:0.001
+ step_lr:0.0001
+ step_lr:0.00001
+ }
+ }
+}
+</pre></div></div></div>
+<div class="section">
+<h3><a name="TrainOneBatch_algorithm"></a>TrainOneBatch algorithm</h3>
+<p>The CNN model is a feed forward model, thus should be configured to use the
[Back-propagation algorithm]({{
BASE_PATH}}/docs/train-one-batch#back-propagation).</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">alg: kBP
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Cluster_setting"></a>Cluster setting</h3>
+<p>The following configuration set a single worker and server for training. <a
class="externalLink"
href="http://singa.incubator.apache.org/docs/frameworks";>Training
frameworks</a> page introduces configurations of a couple of distributed
training frameworks.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">cluster {
+ nworker_groups: 1
+ nserver_groups: 1
+}
+</pre></div></div></div></div>
</div>
</div>
</div>
Modified: websites/staging/singa/trunk/content/docs/code-structure.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/code-structure.html (original)
+++ websites/staging/singa/trunk/content/docs/code-structure.html Wed Sep 2
08:15:15 2015
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache SINGA – Code Structure</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
Modified: websites/staging/singa/trunk/content/docs/communication.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/communication.html (original)
+++ websites/staging/singa/trunk/content/docs/communication.html Wed Sep 2
08:15:15 2015
@@ -1,15 +1,15 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2015-08-17
+ | Generated by Apache Maven Doxia at 2015-09-02
| Rendered using Apache Maven Fluido Skin 1.4
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20150817" />
+ <meta name="Date-Revision-yyyymmdd" content="20150902" />
<meta http-equiv="Content-Language" content="en" />
- <title>Apache SINGA – Communication</title>
+ <title>Apache SINGA – </title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
<link rel="stylesheet" href="../css/site.css" />
<link rel="stylesheet" href="../css/print.css" media="print" />
@@ -189,7 +189,7 @@
Apache SINGA</a>
<span class="divider">/</span>
</li>
- <li class="active ">Communication</li>
+ <li class="active "></li>
@@ -423,14 +423,15 @@
<div id="bodyColumn" class="span10" >
- <div class="section">
-<h2><a name="Communication"></a>Communication</h2>
-<hr />
+ <p>— layout: post title: Communication category : docs</p>
+<div class="section">
+<h2><a name="tags_:_rnn_example"></a>tags : [rnn, example]</h2>
+<p>{% include JB/setup %}</p>
<p>Different messaging libraries has different benefits and drawbacks. For
instance, MPI provides fast message passing between GPUs (using GPUDirect), but
does not support fault-tolerance well. On the contrary, systems using ZeroMQ
can be fault-tolerant, but does not support GPUDirect. The AllReduce function
of MPI is also missing in ZeroMQ which is efficient for data aggregation for
distributed training. In Singa, we provide general messaging APIs for
communication between threads within a process and across processes, and let
users choose the underlying implementation (MPI or ZeroMQ) that meets their
requirements.</p>
<p>Singa’s messaging library consists of two components, namely the
message, and the socket to send and receive messages. <b>Socket</b> refers to a
Singa defined data structure instead of the Linux Socket. We will introduce the
two components in detail with the following figure as an example
architecture.</p>
<p><img src="../images/arch/arch2.png" style="width: 550px" alt="" /> <img
src="../images/arch/comm.png" style="width: 550px" alt="" />
<p><b> Fig.1 - Example physical architecture and network connection</b></p>
-<p>Fig.1 shows an example physical architecture and its network connection. <a
href="architecture.html}">Section-partition server side ParamShard</a> has a
detailed description of the architecture. Each process consists of one main
thread running the stub and multiple background threads running the worker and
server tasks. The stub of the main thread forwards messages among threads . The
worker and server tasks are performed by the background threads.</p>
+<p>Fig.1 shows an example physical architecture and its network connection. <a
class="externalLink"
href="http://singa.incubator.apache.org/docs/architecture.html}";>Section-partition
server side ParamShard</a> has a detailed description of the architecture.
Each process consists of one main thread running the stub and multiple
background threads running the worker and server tasks. The stub of the main
thread forwards messages among threads . The worker and server tasks are
performed by the background threads.</p>
<div class="section">
<h3><a name="Message"></a>Message</h3>
<p><object type="image/svg+xml" style="width: 100px" data="../images/msg.svg">
Not supported </object>
@@ -799,79 +800,50 @@ class SafeQueue{
</pre></div></div>
<p>For inter-process communication, we serialize the message and call
MPI’s send/receive functions to transfer them. All inter-process
connections are setup by MPI at the beginning. Consequently, the Connect and
Bind functions do nothing for both inter-process and intra-process
communication.</p>
<p>MPI’s AllReduce function is efficient for data aggregation in
distributed training. For example, <a class="externalLink"
href="http://arxiv.org/abs/1501.02876";>DeepImage of Baidu</a> uses AllReduce to
aggregate the updates of parameter from all workers. It has similar
architecture as <a href="architecture.html">Fig.2</a>, where every process has
a server group and is connected with all other processes. Hence, we can
implement DeepImage in Singa by simply using MPI’s AllReduce function
for inter-process communication.</p>
-<!-- #### Server socket
+<p>{% comment %}</p></div>
+<div class="section">
+<h4><a name="Server_socket"></a>Server socket</h4>
+<p>Each server has a DEALER socket to communicate with the stub in the main
thread via an <i>in-proc</i> socket. It receives requests issued from workers
and other servers, and forwarded by the ROUTER of the stub. Since the requests
are forwarded by the stub, we can make the location of workers transparent to
server threads. The stub records the locations of workers and servers.</p>
+<p>As explained previously in the [APIs](<a class="externalLink"
href="http://singa.incubator.apache.org{%";>http://singa.incubator.apache.org{%</a>
post_url /docs/2015-03-20-parameter-management %}) for parameter management,
some requests may not be processed immediately but have to be re-queued. For
instance, the Get request cannot be processed if the requested parameter is not
available, i.e., the parameter has not been put into the server’s
ParamShard. The re-queueing operation is implemented sendings the messages to
the ROUTER socket of the stub which treats the message as a newly arrived
request and queues it for processing.</p></div>
+<div class="section">
+<h4><a name="Worker_socket"></a>Worker socket</h4>
+<p>Each worker thread has a DEALER socket to communicate with the stub in the
main thread via an <i>in-proc</i> socket. It sends (Get/Update) requests to the
ROUTER in the stub which forwards the request to (local or remote) processes.
In case of the partition of ParamShard of worker side, it may also transfer
data with other workers via the DEALER socket. Again, the location of the other
side (a server or worker) of the communication is transparent to the worker.
The stub handles the addressing.</p>
+<p>PMClient executes the training logic, during which it generates GET and
UPDATE requests. A request received at the worker’s main thread contains
ID of the PMClient instance. The worker determines which server to send the
request based on its content, then sends it via the corresponding socket.
Response messages received from any of the server socket are forwarded to the
in-proc ROUTER socket. Since each response header contains the PMClient ID, it
is routed to the correct instance.</p></div>
+<div class="section">
+<h4><a name="Stub_sockets"></a>Stub sockets</h4>
+<div class="section">
+<h5><a name="ROUTER_socket"></a>ROUTER socket</h5>
+<p>The main thread has a ROUTER socket to communicate with background
threads.</p>
+<p>It forwards the requests from workers to background servers. There can be
multiple servers.If all servers maintain the same (sub) ParamShard, then the
request can be forwarded to any of them. Load-balance (like round-robin) can be
implemented in the stub to improve the performance. If each server maintains a
sub-set of the local ParamShard, then the stub forwards each request to the
corresponding server. It also forwards the synchronization requests from remote
servers to local servers in the same way.</p>
+<p>In the case of neural network partition (i.e., model partition), neighbor
layers would transfer data with each other. Hence, the ROUTER would forwards
data transfer requests from one worker to other worker. The stub looks up the
location table to decide where to forward each request.</p></div>
+<div class="section">
+<h5><a name="DEALER_sockets"></a>DEALER sockets</h5>
+<p>The main thread has multiple DEALER sockets to communicate with other
processes, one socket per process. Two processes are connected if one of the
following cases exists:</p>
+
+<ul>
+
+<li>one worker group spans across the two processes;</li>
+
+<li>two connected server groups are separated in the two processes;</li>
+
+<li>workers and the subscribed servers are separated in the two processes.</li>
+</ul>
+<p>All messages in SINGA are of multi-frame ZeroMQ format. The figure above
demonstrates different types of messages exchanged in the system.</p>
-Each server has a DEALER socket to communicate with the stub in the main
-thread via an _in-proc_ socket. It receives requests issued from workers and
-other servers, and forwarded by the ROUTER of the stub. Since the requests are
forwarded by the
-stub, we can make the location of workers transparent to server threads. The
-stub records the locations of workers and servers.
-
-As explained previously in the
-[APIs]({{ BASE_PATH }}{% post_url /docs/2015-03-20-parameter-management %})
-for parameter management, some requests may
-not be processed immediately but have to be re-queued. For instance, the Get
-request cannot be processed if the requested parameter is not available, i.e.,
-the parameter has not been put into the server's ParamShard. The re-queueing
-operation is implemented sendings the messages to the ROUTER
-socket of the stub which treats the message as a newly arrived request
-and queues it for processing.
-
-#### Worker socket
-
-Each worker thread has a DEALER socket to communicate with the stub in the main
-thread via an _in-proc_ socket. It sends (Get/Update) requests to the ROUTER in
-the stub which forwards the request to (local or remote) processes. In case of
-the partition of ParamShard of worker side, it may also transfer data with
other
-workers via the DEALER socket. Again, the location of the other side (a server
-or worker) of the communication is transparent to the worker. The stub handles
-the addressing.
-
-PMClient executes the training logic, during which it generates GET and UPDATE
-requests. A request received at the worker's main thread contains ID of the
-PMClient instance. The worker determines which server to send the request based
-on its content, then sends it via the corresponding socket. Response messages
-received from any of the server socket are forwarded to the in-proc ROUTER
-socket. Since each response header contains the PMClient ID, it is routed to
-the correct instance.
-
-#### Stub sockets
-
-##### ROUTER socket
-The main thread has a ROUTER socket to communicate with background threads.
-
-It forwards the requests from workers to background servers. There can be
-multiple servers.If all servers maintain the same (sub) ParamShard, then the
-request can be forwarded to any of them. Load-balance (like round-robin) can be
-implemented in the stub to improve the performance. If each server maintains a
-sub-set of the local ParamShard, then the stub forwards each request to the
-corresponding server. It also forwards the synchronization requests from
-remote servers to local servers in the same way.
-
-In the case of neural network partition (i.e., model partition), neighbor
-layers would transfer data with each other. Hence, the ROUTER would forwards
-data transfer requests from one worker to other worker. The stub looks up the
-location table to decide where to forward each request.
-
-##### DEALER sockets
-
-The main thread has multiple DEALER sockets to communicate with other
-processes, one socket per process. Two processes are connected if one of the
-following cases exists:
-
- * one worker group spans across the two processes;
- * two connected server groups are separated in the two processes;
- * workers and the subscribed servers are separated in the two processes.
-
-
-All messages in SINGA are of multi-frame ZeroMQ format. The figure above
demonstrates different types of messages exchanged in the system.
-
- 1. Requests generated by PMClient consist of the parameter content (which
could be empty), followed by the parameter ID (key) and the request type
(GET/PUT/REQUEST). Responses received by PMClient are also of this format.
- 2. Messages received by the worker's main thread from PMClient instances
contain another frame identifying the PMClient connection (or PMClient ID).
- 3. Requests originating form a worker and arriving at the server contain
another frame identifying the worker's connection (or Worker ID).
- 4. Requests originating from another server and arriving at the server have
the same format as (3), but the first frame identifies the server connection
(or Server ID).
- 5. After a PMServer processes a request, it generates a message with the
format similar to (3) but with extra frame indicating if the message is to be
routed back to a worker (a response message) or to route to another server (a
SYNC request).
- 6. When a request is re-queued, the PMServer generates a message and sends
it directly to the server's front-end socket. The re-queued request seen by the
server's main thread consists of all the frames in (3), followed by a REQUEUED
frame, and finally by another frame generated by the ROUTER socket identifying
connection from the PMServer instance. The main thread then strips off these
additional two frames before forwarding it to another PMServer instance like
another ordinary request. --></div></div></div>
+<ol style="list-style-type: decimal">
+
+<li>Requests generated by PMClient consist of the parameter content (which
could be empty), followed by the parameter ID (key) and the request type
(GET/PUT/REQUEST). Responses received by PMClient are also of this format.</li>
+
+<li>Messages received by the worker’s main thread from PMClient
instances contain another frame identifying the PMClient connection (or
PMClient ID).</li>
+
+<li>Requests originating form a worker and arriving at the server contain
another frame identifying the worker’s connection (or Worker ID).</li>
+
+<li>Requests originating from another server and arriving at the server have
the same format as (3), but the first frame identifies the server connection
(or Server ID).</li>
+
+<li>After a PMServer processes a request, it generates a message with the
format similar to (3) but with extra frame indicating if the message is to be
routed back to a worker (a response message) or to route to another server (a
SYNC request).</li>
+
+<li>When a request is re-queued, the PMServer generates a message and sends it
directly to the server’s front-end socket. The re-queued request seen by
the server’s main thread consists of all the frames in (3), followed by
a REQUEUED frame, and finally by another frame generated by the ROUTER socket
identifying connection from the PMServer instance. The main thread then strips
off these additional two frames before forwarding it to another PMServer
instance like another ordinary request. {% endcomment %}</li>
+</ol></div></div></div></div>
</div>
</div>
</div>
Added: websites/staging/singa/trunk/content/docs/data.html
==============================================================================
(empty)
![http://singa.incubator.apache.org/assets/image/logical.png"](http://singa.incubator.apache.org/assets/image/logical.png"){kind=link}
![http://singa.incubator.apache.org/assets/image/cnn-example.png"](http://singa.incubator.apache.org/assets/image/cnn-example.png"){kind=link}