Author: buildbot
Date: Sat Sep 26 12:10:18 2015
New Revision: 966886
Log:
Staging update by buildbot for singa
Modified:
websites/staging/singa/trunk/content/ (props changed)
websites/staging/singa/trunk/content/docs/data.html
websites/staging/singa/trunk/content/docs/distributed-training.html
websites/staging/singa/trunk/content/docs/examples.html
websites/staging/singa/trunk/content/docs/frameworks.html
websites/staging/singa/trunk/content/docs/model-config.html
websites/staging/singa/trunk/content/docs/programming-guide.html
websites/staging/singa/trunk/content/docs/train-one-batch.html
Propchange: websites/staging/singa/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Sat Sep 26 12:10:18 2015
@@ -1 +1 @@
-1705429
+1705434
Modified: websites/staging/singa/trunk/content/docs/data.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/data.html (original)
+++ websites/staging/singa/trunk/content/docs/data.html Sat Sep 26 12:10:18 2015
@@ -489,6 +489,7 @@
<div id="bodyColumn" class="span10" >
<h1>Data Preparation</h1>
+<hr />
<p>To submit a training job, users need to convert raw data (e.g., images,
text documents) into SINGA recognizable <a
href="../api/classsinga_1_1Record.html">Record</a>s. SINGA uses <a
href="layer#data-layers">data layers</a> to load these records into memory and
uses <a href="layer#parser-layers">parser layers</a> to parse features (e.g.,
image pixels and labels) from these <tt>Record</tt>s. <tt>Record</tt>s could be
stored in a file, a database, or HDFS, as long as there is a corresponding <a
href="../api/classsinga_1_1DataLayer.html">DataLayer</a>.</p>
<div class="section">
<h2><a name="DataShard"></a>DataShard</h2>
Modified: websites/staging/singa/trunk/content/docs/distributed-training.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/distributed-training.html
(original)
+++ websites/staging/singa/trunk/content/docs/distributed-training.html Sat Sep
26 12:10:18 2015
@@ -489,6 +489,7 @@
<div id="bodyColumn" class="span10" >
<h1>Distributed Training</h1>
+<hr />
<p>SINGA is designed for distributed training of large deep learning models
with huge amount of training data.</p>
<p>Here we introduce distrbuted SINGA in following aspects:</p>
Modified: websites/staging/singa/trunk/content/docs/examples.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/examples.html (original)
+++ websites/staging/singa/trunk/content/docs/examples.html Sat Sep 26 12:10:18
2015
@@ -489,6 +489,7 @@
<div id="bodyColumn" class="span10" >
<h1>Example Models</h1>
+<hr />
<p>Different models are provided as examples to help users get familiar with
SINGA. <a href="neural-net.html">Neural Network</a> gives details on the models
that are supported by SINGA.</p>
<div class="section">
<div class="section">
Modified: websites/staging/singa/trunk/content/docs/frameworks.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/frameworks.html (original)
+++ websites/staging/singa/trunk/content/docs/frameworks.html Sat Sep 26
12:10:18 2015
@@ -489,9 +489,9 @@
<div id="bodyColumn" class="span10" >
<h1>Distributed Training Framework</h1>
+<hr />
<div class="section">
-<div class="section">
-<h3><a name="Cluster_Topology_Configuration"></a>Cluster Topology
Configuration</h3>
+<h2><a name="Cluster_Topology_Configuration"></a>Cluster Topology
Configuration</h2>
<p>Here we describe how to configure SINGA’s cluster topology to
support different distributed training frameworks. The cluster topology is
configured in the <tt>cluster</tt> field in <tt>JobProto</tt>. The
<tt>cluster</tt> is of type <tt>ClusterProto</tt>:</p>
<div class="source">
@@ -520,12 +520,12 @@
<li><tt>server_worker_separate</tt>: separate servers and workers in
different processes.</li>
</ul></div>
<div class="section">
-<h3><a name="Different_Training_Frameworks"></a>Different Training
Frameworks</h3>
+<h2><a name="Different_Training_Frameworks"></a>Different Training
Frameworks</h2>
<p>In SINGA, worker groups run asynchronously and workers within one group run
synchronously. Users can leverage this general design to run both
<b>synchronous</b> and <b>asynchronous</b> training frameworks. Here we
illustrate how to configure popular distributed training frameworks in
SINGA.</p>
<p><img src="../images/frameworks.png" style="width: 800px" alt="" />
<p><b> Fig.1 - Training frameworks in SINGA</b></p>
<div class="section">
-<h4><a name="Sandblaster"></a>Sandblaster</h4>
+<h3><a name="Sandblaster"></a>Sandblaster</h3>
<p>This is a <b>synchronous</b> framework used by Google Brain. Fig.2(a) shows
the Sandblaster framework implemented in SINGA. Its configuration is as
follows:</p>
<div class="source">
@@ -539,7 +539,7 @@
</pre></div></div>
<p>A single server group is launched to handle all requests from workers. A
worker computes on its partition of the model, and only communicates with
servers handling related parameters.</p></div>
<div class="section">
-<h4><a name="AllReduce"></a>AllReduce</h4>
+<h3><a name="AllReduce"></a>AllReduce</h3>
<p>This is a <b>synchronous</b> framework used by Baidu’s DeepImage.
Fig.2(b) shows the AllReduce framework implemented in SINGA. Its configuration
is as follows:</p>
<div class="source">
@@ -553,7 +553,7 @@
</pre></div></div>
<p>We bind each worker with a server on the same node, so that each node is
responsible for maintaining a partition of parameters and collecting updates
from all other nodes.</p></div>
<div class="section">
-<h4><a name="Downpour"></a>Downpour</h4>
+<h3><a name="Downpour"></a>Downpour</h3>
<p>This is a <b>asynchronous</b> framework used by Google Brain. Fig.2(c)
shows the Downpour framework implemented in SINGA. Its configuration is as
follows:</p>
<div class="source">
@@ -567,7 +567,7 @@
</pre></div></div>
<p>Similar to the synchronous Sandblaster, all workers send requests to a
global server group. We divide workers into several worker groups, each running
independently and working on parameters from the last <i>update</i>
response.</p></div>
<div class="section">
-<h4><a name="Distributed_Hogwild"></a>Distributed Hogwild</h4>
+<h3><a name="Distributed_Hogwild"></a>Distributed Hogwild</h3>
<p>This is a <b>asynchronous</b> framework used by Caffe. Fig.2(d) shows the
Distributed Hogwild framework implemented in SINGA. Its configuration is as
follows:</p>
<div class="source">
@@ -579,7 +579,7 @@
server_worker_separate: false
}
</pre></div></div>
-<p>Each node contains a complete server group and a complete worker group.
Parameter updates are done locally, so that communication cost during each
training step is minimized. However, the server group must periodically
synchronize with neighboring groups to improve the training
convergence.</p></div></div></div>
+<p>Each node contains a complete server group and a complete worker group.
Parameter updates are done locally, so that communication cost during each
training step is minimized. However, the server group must periodically
synchronize with neighboring groups to improve the training
convergence.</p></div></div>
</div>
</div>
</div>
Modified: websites/staging/singa/trunk/content/docs/model-config.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/model-config.html (original)
+++ websites/staging/singa/trunk/content/docs/model-config.html Sat Sep 26
12:10:18 2015
@@ -489,12 +489,12 @@
<div id="bodyColumn" class="span10" >
<h1>Model Configuration</h1>
+<hr />
<p>SINGA uses the stochastic gradient descent (SGD) algorithm to train
parameters of deep learning models. For each SGD iteration, there is a <a
href="architecture.html">Worker</a> computing gradients of parameters from the
NeuralNet and a <a href="">Updater</a> updating parameter values based on
gradients. Hence the model configuration mainly consists these three parts. We
will introduce the NeuralNet, Worker and Updater in the following paragraphs
and describe the configurations for them. All model configuration is specified
in the model.conf file in the user provided workspace folder. E.g., the <a
class="externalLink"
href="https://github.com/apache/incubator-singa/tree/master/examples/cifar10";>cifar10
example folder</a> has a model.conf file.</p>
<div class="section">
+<h2><a name="NeuralNet"></a>NeuralNet</h2>
<div class="section">
-<h3><a name="NeuralNet"></a>NeuralNet</h3>
-<div class="section">
-<h4><a name="Uniform_model_neuralnet_representation"></a>Uniform model
(neuralnet) representation</h4>
+<h3><a name="Uniform_model_neuralnet_representation"></a>Uniform model
(neuralnet) representation</h3>
<p><img src="../images/model-categorization.png" style="width: 400px" alt=""
/> Fig. 1: Deep learning model categorization</img></p>
<p>Many deep learning models have being proposed. Fig. 1 is a categorization
of popular deep learning models based on the layer connections. The <a
class="externalLink"
href="https://github.com/apache/incubator-singa/blob/master/include/neuralnet/neuralnet.h";>NeuralNet</a>
abstraction of SINGA consists of multiple directly connected layers. This
abstraction is able to represent models from all the three categorizations.</p>
@@ -540,7 +540,7 @@ LayerProto layer_proto_; // layer confi
<div class="source"><pre class="prettyprint">Param weight;
</pre></div></div></div>
<div class="section">
-<h4><a name="Configure_the_structure_of_a_NeuralNet_instance"></a>Configure
the structure of a NeuralNet instance</h4>
+<h3><a name="Configure_the_structure_of_a_NeuralNet_instance"></a>Configure
the structure of a NeuralNet instance</h3>
<p>To train a deep learning model, the first step is to write the
configurations for the model structure, i.e., the layers and connections for
the NeuralNet. Like <a class="externalLink"
href="http://caffe.berkeleyvision.org/";>Caffe</a>, we use the <a
class="externalLink"
href="https://developers.google.com/protocol-buffers/";>Google Protocol
Buffer</a> to define the configuration protocol. The <a class="externalLink"
href="https://github.com/apache/incubator-singa/blob/master/src/proto/model.proto";>NetProto</a>
specifies the configuration fields for a NeuralNet instance,</p>
<p>message NetProto { repeated LayerProto layer = 1; … }</p>
<p>The configuration is then</p>
@@ -604,7 +604,7 @@ layer {
<p>Fig. 4 shows the model structure corresponding to the neural network
configuration for the <a class="externalLink"
href="https://github.com/apache/incubator-singa/blob/master/examples/cifar10/model.conf";>deep
convolutional model</a>.</p>
<p><img src="../images/dcnn-cifar10.png" style="width: 200px" alt="" /> Fig.
4: Neural network structure for the example DCNN model
(cifar10)</img></p></div></div>
<div class="section">
-<h3><a name="Worker"></a>Worker</h3>
+<h2><a name="Worker"></a>Worker</h2>
<p>At the beginning, the Work will initialize the values of Param instances of
each layer either randomly (according to user configured distribution) or
loading from a <a href="">checkpoint file</a>. For each training iteration, the
worker visits layers of the neural network to compute gradients of Param
instances of each layer. Corresponding to the three categories of models, there
are three different algorithm to compute the gradients of a neural network.</p>
<ol style="list-style-type: decimal">
@@ -668,7 +668,7 @@ layer {
</pre></div></div>
<p>All <a href="">Layer implementations</a> must implement the above two
functions.</p></div>
<div class="section">
-<h3><a name="Updater"></a>Updater</h3>
+<h2><a name="Updater"></a>Updater</h2>
<p>Once the gradients of parameters are computed, the Updater will update
parameter values. There are many SGD variants for updating parameters, like <a
class="externalLink" href="http://arxiv.org/pdf/1212.5701v1.pdf";>AdaDelta</a>,
<a class="externalLink"
href="http://www.magicbroom.info/Papers/DuchiHaSi10.pdf";>AdaGrad</a>, <a
class="externalLink"
href="http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf";>RMSProp</a>,
<a class="externalLink"
href="http://scholar.google.com/citations?view_op=view_citation&hl=en&user=DJ8Ep8YAAAAJ&citation_for_view=DJ8Ep8YAAAAJ:hkOj_22Ku90C";>Nesterov</a>
and SGD with momentum. The core functions of the Updater is</p>
<div class="source">
@@ -725,7 +725,7 @@ float GetLearningRate(int step);
}
</pre></div></div></div>
<div class="section">
-<h3><a name="Other_model_configuration_fields"></a>Other model configuration
fields</h3>
+<h2><a name="Other_model_configuration_fields"></a>Other model configuration
fields</h2>
<p>Some other important configuration fields for training a deep learning
model is listed:</p>
<div class="source">
@@ -741,7 +741,7 @@ optional int32 checkpoint_frequency = 34
// checkpoint path
optional bool resume = 36 [default = false];
</pre></div></div>
-<p>The pages of <a href="checkpoint.html">checkpoint and restore</a> has more
details on related fields.</p></div></div>
+<p>The pages of <a href="checkpoint.html">checkpoint and restore</a> has more
details on related fields.</p></div>
</div>
</div>
</div>
Modified: websites/staging/singa/trunk/content/docs/programming-guide.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/programming-guide.html (original)
+++ websites/staging/singa/trunk/content/docs/programming-guide.html Sat Sep 26
12:10:18 2015
@@ -491,6 +491,7 @@
<div id="bodyColumn" class="span10" >
<h1>Programming Guide</h1>
+<hr />
<p>To submit a training job, users must provide the configuration of the four
components shown in Figure 1:</p>
<ul>
Modified: websites/staging/singa/trunk/content/docs/train-one-batch.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/train-one-batch.html (original)
+++ websites/staging/singa/trunk/content/docs/train-one-batch.html Sat Sep 26
12:10:18 2015
@@ -489,6 +489,7 @@
<div id="bodyColumn" class="span10" >
<h1>Train-One-Batch</h1>
+<hr />
<p>For each SGD iteration, every worker calls the <tt>TrainOneBatch</tt>
function to compute gradients of parameters associated with local layers (i.e.,
layers dispatched to it). SINGA has implemented two algorithms for the
<tt>TrainOneBatch</tt> function. Users select the corresponding algorithm for
their model in the configuration.</p>
<div class="section">
<h2><a name="Basic_user_guide"></a>Basic user guide</h2>
