Added: websites/staging/singa/trunk/content/v0.3.0/updater.html ============================================================================== --- websites/staging/singa/trunk/content/v0.3.0/updater.html (added) +++ websites/staging/singa/trunk/content/v0.3.0/updater.html Wed Apr 20 05:12:03 2016 @@ -0,0 +1,650 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-20 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160420" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – Updater</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../docs/index.html" title="English">English</a> +</li> + <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">Updater</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>Updater</h1> +<hr /> +<p>Every server in SINGA has an <a href="../api/classsinga_1_1Updater.html">Updater</a> instance that updates parameters based on gradients. In this page, the <i>Basic user guide</i> describes the configuration of an updater. The <i>Advanced user guide</i> present details on how to implement a new updater and a new learning rate changing method.</p> +<div class="section"> +<h2><a name="Basic_user_guide"></a>Basic user guide</h2> +<p>There are many different parameter updating protocols (i.e., subclasses of <tt>Updater</tt>). They share some configuration fields like</p> + +<ul> + +<li><tt>type</tt>, an integer for identifying an updater;</li> + +<li><tt>learning_rate</tt>, configuration for the <a href="../api/classsinga_1_1LRGenerator.html">LRGenerator</a> which controls the learning rate.</li> + +<li><tt>weight_decay</tt>, the co-efficient for <a class="externalLink" href="http://deeplearning.net/tutorial/gettingstarted.html#regularization";>L2 * regularization</a>.</li> + +<li><a class="externalLink" href="http://ufldl.stanford.edu/tutorial/supervised/OptimizationStochasticGradientDescent/";>momentum</a>.</li> +</ul> +<p>If you are not familiar with the above terms, you can get their meanings in <a class="externalLink" href="http://cs231n.github.io/neural-networks-3/#update";>this page provided by Karpathy</a>.</p> +<div class="section"> +<h3><a name="Configuration_of_built-in_updater_classes"></a>Configuration of built-in updater classes</h3> +<div class="section"> +<h4><a name="Updater"></a>Updater</h4> +<p>The base <tt>Updater</tt> implements the <a class="externalLink" href="http://cs231n.github.io/neural-networks-3/#sgd";>vanilla SGD algorithm</a>. Its configuration type is <tt>kSGD</tt>. Users need to configure at least the <tt>learning_rate</tt> field. <tt>momentum</tt> and <tt>weight_decay</tt> are optional fields.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">updater{ + type: kSGD + momentum: float + weight_decay: float + learning_rate { + ... + } +} +</pre></div></div></div> +<div class="section"> +<h4><a name="AdaGradUpdater"></a>AdaGradUpdater</h4> +<p>It inherits the base <tt>Updater</tt> to implement the <a class="externalLink" href="http://www.magicbroom.info/Papers/DuchiHaSi10.pdf";>AdaGrad</a> algorithm. Its type is <tt>kAdaGrad</tt>. <tt>AdaGradUpdater</tt> is configured similar to <tt>Updater</tt> except that <tt>momentum</tt> is not used.</p></div> +<div class="section"> +<h4><a name="NesterovUpdater"></a>NesterovUpdater</h4> +<p>It inherits the base <tt>Updater</tt> to implements the <a class="externalLink" href="http://arxiv.org/pdf/1212.0901v2.pdf";>Nesterov</a> (section 3.5) updating protocol. Its type is <tt>kNesterov</tt>. <tt>learning_rate</tt> and <tt>momentum</tt> must be configured. <tt>weight_decay</tt> is an optional configuration field.</p></div> +<div class="section"> +<h4><a name="RMSPropUpdater"></a>RMSPropUpdater</h4> +<p>It inherits the base <tt>Updater</tt> to implements the <a class="externalLink" href="http://cs231n.github.io/neural-networks-3/#sgd";>RMSProp algorithm</a> proposed by <a class="externalLink" href="http://www.cs.toronto.edu/%7Etijmen/csc321/slides/lecture_slides_lec6.pdf";>Hinton</a>(slide 29). Its type is <tt>kRMSProp</tt>.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">updater { + type: kRMSProp + rmsprop_conf { + rho: float # [0,1] + } +} +</pre></div></div></div> +<div class="section"> +<h4><a name="AdaDeltaUpdater"></a>AdaDeltaUpdater</h4> +<p>It inherits the base <tt>Updater</tt> to implements the <a class="externalLink" href="http://arxiv.org/abs/1212.5701";>AdaDelta</a> updating algorithm. Its type is <tt>kAdaDelta</tt>.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">updater { + type: kAdaDelta + adadelta_conf { + rho: float # [0,1] + } +} +</pre></div></div></div> +<div class="section"> +<h4><a name="Adam"></a>Adam</h4> +<p>It inherits the base <tt>Updater</tt> to implements the <a class="externalLink" href="http://arxiv.org/pdf/1412.6980.pdf";>Adam</a> updating algorithm. Its type is <tt>kAdam</tt>. <tt>beta1</tt> and <tt>beta2</tt> is floats, 0 < <tt>beta</tt> < 1, generally close to 1.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">updater { + type: kAdam + adam_conf { + beta1: float # [0,1] + beta2: float # [0,1] + } +} +</pre></div></div></div> +<div class="section"> +<h4><a name="AdaMax"></a>AdaMax</h4> +<p>It inherits the base <tt>Updater</tt> to implements the <a class="externalLink" href="http://arxiv.org/pdf/1412.6980.pdf";>AdaMax</a> updating algorithm. Its type is <tt>kAdamMax</tt>. <tt>beta1</tt> and <tt>beta2</tt> is floats, 0 < <tt>beta</tt> < 1, generally close to 1.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">updater { + type: kAdamMax + adammax_conf { + beta1: float # [0,1] + beta2: float # [0,1] + } +} +</pre></div></div></div></div> +<div class="section"> +<h3><a name="Configuration_of_learning_rate"></a>Configuration of learning rate</h3> +<p>The <tt>learning_rate</tt> field is configured as,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + type: ChangeMethod + base_lr: float # base/initial learning rate + ... # fields to a specific changing method +} +</pre></div></div> +<p>The common fields include <tt>type</tt> and <tt>base_lr</tt>. SINGA provides the following <tt>ChangeMethod</tt>s.</p> +<div class="section"> +<h4><a name="kFixed"></a>kFixed</h4> +<p>The <tt>base_lr</tt> is used for all steps.</p></div> +<div class="section"> +<h4><a name="kLinear"></a>kLinear</h4> +<p>The updater should be configured like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + base_lr: float + linear_conf { + freq: int + final_lr: float + } +} +</pre></div></div> +<p>Linear interpolation is used to change the learning rate,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">lr = (1 - step / freq) * base_lr + (step / freq) * final_lr +</pre></div></div></div> +<div class="section"> +<h4><a name="kExponential"></a>kExponential</h4> +<p>The udapter should be configured like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + base_lr: float + exponential_conf { + freq: int + } +} +</pre></div></div> +<p>The learning rate for <tt>step</tt> is</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">lr = base_lr / 2^(step / freq) +</pre></div></div></div> +<div class="section"> +<h4><a name="kInverseT"></a>kInverseT</h4> +<p>The updater should be configured like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + base_lr: float + inverset_conf { + final_lr: float + } +} +</pre></div></div> +<p>The learning rate for <tt>step</tt> is</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">lr = base_lr / (1 + step / final_lr) +</pre></div></div></div> +<div class="section"> +<h4><a name="kInverse"></a>kInverse</h4> +<p>The updater should be configured like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + base_lr: float + inverse_conf { + gamma: float + pow: float + } +} +</pre></div></div> +<p>The learning rate for <tt>step</tt> is</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">lr = base_lr * (1 + gamma * setp)^(-pow) +</pre></div></div></div> +<div class="section"> +<h4><a name="kStep"></a>kStep</h4> +<p>The updater should be configured like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + base_lr : float + step_conf { + change_freq: int + gamma: float + } +} +</pre></div></div> +<p>The learning rate for <tt>step</tt> is</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">lr = base_lr * gamma^ (step / change_freq) +</pre></div></div></div> +<div class="section"> +<h4><a name="kFixedStep"></a>kFixedStep</h4> +<p>The updater should be configured like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + fixedstep_conf { + step: int + step_lr: float + + step: int + step_lr: float + + ... + } +} +</pre></div></div> +<p>Denote the i-th tuple as (step[i], step_lr[i]), then the learning rate for <tt>step</tt> is,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">step_lr[k] +</pre></div></div> +<p>where step[k] is the smallest number that is larger than <tt>step</tt>.</p></div></div></div> +<div class="section"> +<h2><a name="Advanced_user_guide"></a>Advanced user guide</h2> +<div class="section"> +<h3><a name="Implementing_a_new_Updater_subclass"></a>Implementing a new Updater subclass</h3> +<p>The base Updater class has one virtual function,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">class Updater{ + public: + virtual void Update(int step, Param* param, float grad_scale = 1.0f) = 0; + + protected: + UpdaterProto proto_; + LRGenerator lr_gen_; +}; +</pre></div></div> +<p>It updates the values of the <tt>param</tt> based on its gradients. The <tt>step</tt> argument is for deciding the learning rate which may change through time (step). <tt>grad_scale</tt> scales the original gradient values. This function is called by servers once it receives all gradients for the same <tt>Param</tt> object.</p> +<p>To implement a new Updater subclass, users must override the <tt>Update</tt> function.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">class FooUpdater : public Updater { + void Update(int step, Param* param, float grad_scale = 1.0f) override; +}; +</pre></div></div> +<p>Configuration of this new updater can be declared similar to that of a new layer,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in user.proto +FooUpdaterProto { + optional int32 c = 1; +} + +extend UpdaterProto { + optional FooUpdaterProto fooupdater_conf= 101; +} +</pre></div></div> +<p>The new updater should be registered in the <a href="programming-guide.html">main function</a></p> + +<div class="source"> +<div class="source"><pre class="prettyprint">driver.RegisterUpdater<FooUpdater>("FooUpdater"); +</pre></div></div> +<p>Users can then configure the job as</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in job.conf +updater { + user_type: "FooUpdater" # must use user_type with the same string identifier as the one used for registration + fooupdater_conf { + c : 20; + } +} +</pre></div></div></div> +<div class="section"> +<h3><a name="Implementing_a_new_LRGenerator_subclass"></a>Implementing a new LRGenerator subclass</h3> +<p>The base <tt>LRGenerator</tt> is declared as,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">virtual float Get(int step); +</pre></div></div> +<p>To implement a subclass, e.g., <tt>FooLRGen</tt>, users should declare it like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">class FooLRGen : public LRGenerator { + public: + float Get(int step) override; +}; +</pre></div></div> +<p>Configuration of <tt>FooLRGen</tt> can be defined using a protocol message,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in user.proto +message FooLRProto { + ... +} + +extend LRGenProto { + optional FooLRProto foolr_conf = 101; +} +</pre></div></div> +<p>The configuration is then like,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">learning_rate { + user_type : "FooLR" # must use user_type with the same string identifier as the one used for registration + base_lr: float + foolr_conf { + ... + } +} +</pre></div></div> +<p>Users have to register this subclass in the main function,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"> driver.RegisterLRGenerator<FooLRGen, std::string>("FooLR") +</pre></div></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html>
Added: websites/staging/singa/trunk/content/v0.3.0/zh/checkpoint.html ============================================================================== --- websites/staging/singa/trunk/content/v0.3.0/zh/checkpoint.html (added) +++ websites/staging/singa/trunk/content/v0.3.0/zh/checkpoint.html Wed Apr 20 05:12:03 2016 @@ -0,0 +1,412 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-20 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160420" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – CheckPoint</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../../docs/index.html" title="English">English</a> +</li> + <li> <a href="../../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">CheckPoint</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>CheckPoint</h1> +<hr /> +<p>SINGA checkpoints model parameters onto disk periodically according to user configured frequency. By checkpointing model parameters, we can</p> + +<ol style="list-style-type: decimal"> + +<li> +<p>resume the training from the last checkpointing. For example, if the program crashes before finishing all training steps, we can continue the training using checkpoint files.</p></li> + +<li> +<p>use them to initialize a similar model. For example, the parameters from training a RBM model can be used to initialize a <a href="rbm.html">deep auto-encoder</a> model.</p></li> +</ol> +<div class="section"> +<h2><a name="Configuration"></a>Configuration</h2> +<p>Checkpointing is controlled by two configuration fields:</p> + +<ul> + +<li><tt>checkpoint_after</tt>, start checkpointing after this number of training steps,</li> + +<li><tt>checkpoint_freq</tt>, frequency of doing checkpointing.</li> +</ul> +<p>For example,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># job.conf +checkpoint_after: 100 +checkpoint_frequency: 300 +... +</pre></div></div> +<p>Checkpointing files are located at <i>WORKSPACE/checkpoint/stepSTEP-workerWORKERID</i>. <i>WORKSPACE</i> is configured in</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">cluster { + workspace: +} +</pre></div></div> +<p>For the above configuration, after training for 700 steps, there would be two checkpointing files,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">step400-worker0 +step700-worker0 +</pre></div></div></div> +<div class="section"> +<h2><a name="Application_-_resuming_training"></a>Application - resuming training</h2> +<p>We can resume the training from the last checkpoint (i.e., step 700) by,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">./bin/singa-run.sh -conf JOB_CONF -resume +</pre></div></div> +<p>There is no change to the job configuration.</p></div> +<div class="section"> +<h2><a name="Application_-_model_initialization"></a>Application - model initialization</h2> +<p>We can also use the checkpointing file from step 400 to initialize a new model by configuring the new job as,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># job.conf +checkpoint : "WORKSPACE/checkpoint/step400-worker0" +... +</pre></div></div> +<p>If there are multiple checkpointing files for the same snapshot due to model partitioning, all the checkpointing files should be added,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># job.conf +checkpoint : "WORKSPACE/checkpoint/step400-worker0" +checkpoint : "WORKSPACE/checkpoint/step400-worker1" +... +</pre></div></div> +<p>The training command is the same as starting a new job,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">./bin/singa-run.sh -conf JOB_CONF +</pre></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html> Added: websites/staging/singa/trunk/content/v0.3.0/zh/cnn.html ============================================================================== --- websites/staging/singa/trunk/content/v0.3.0/zh/cnn.html (added) +++ websites/staging/singa/trunk/content/v0.3.0/zh/cnn.html Wed Apr 20 05:12:03 2016 @@ -0,0 +1,563 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-20 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160420" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – CNN Example</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../../docs/index.html" title="English">English</a> +</li> + <li> <a href="../../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">CNN Example</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>CNN Example</h1> +<hr /> +<p>Convolutional neural network (CNN) is a type of feed-forward artificial neural network widely used for image and video classification. In this example, we will use a deep CNN model to do image classification for the <a class="externalLink" href="http://www.cs.toronto.edu/~kriz/cifar.html";>CIFAR10 dataset</a>.</p> +<div class="section"> +<h2><a name="Running_instructions"></a>Running instructions</h2> +<p>Please refer to the <a href="installation.html">installation</a> page for instructions on building SINGA, and the <a href="quick-start.html">quick start</a> for instructions on starting zookeeper.</p> +<p>We have provided scripts for preparing the training and test dataset in <i>examples/cifar10/</i>.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in examples/cifar10 +$ cp Makefile.example Makefile +$ make download +$ make create +</pre></div></div> +<div class="section"> +<h3><a name="Training_on_CPU"></a>Training on CPU</h3> +<p>We can start the training by</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">./bin/singa-run.sh -conf examples/cifar10/job.conf +</pre></div></div> +<p>You should see output like</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">Record job information to /tmp/singa-log/job-info/job-2-20150817-055601 +Executing : ./singa -conf /xxx/incubator-singa/examples/cifar10/job.conf -singa_conf /xxx/incubator-singa/conf/singa.conf -singa_job 2 +E0817 06:56:18.868259 33849 cluster.cc:51] proc #0 -> 192.168.5.128:49152 (pid = 33849) +E0817 06:56:18.928452 33871 server.cc:36] Server (group = 0, id = 0) start +E0817 06:56:18.928469 33872 worker.cc:134] Worker (group = 0, id = 0) start +E0817 06:57:13.657302 33849 trainer.cc:373] Test step-0, loss : 2.302588, accuracy : 0.077900 +E0817 06:57:17.626708 33849 trainer.cc:373] Train step-0, loss : 2.302578, accuracy : 0.062500 +E0817 06:57:24.142645 33849 trainer.cc:373] Train step-30, loss : 2.302404, accuracy : 0.131250 +E0817 06:57:30.813354 33849 trainer.cc:373] Train step-60, loss : 2.302248, accuracy : 0.156250 +E0817 06:57:37.556655 33849 trainer.cc:373] Train step-90, loss : 2.301849, accuracy : 0.175000 +E0817 06:57:44.971276 33849 trainer.cc:373] Train step-120, loss : 2.301077, accuracy : 0.137500 +E0817 06:57:51.801949 33849 trainer.cc:373] Train step-150, loss : 2.300410, accuracy : 0.135417 +E0817 06:57:58.682281 33849 trainer.cc:373] Train step-180, loss : 2.300067, accuracy : 0.127083 +E0817 06:58:05.578366 33849 trainer.cc:373] Train step-210, loss : 2.300143, accuracy : 0.154167 +E0817 06:58:12.518497 33849 trainer.cc:373] Train step-240, loss : 2.295912, accuracy : 0.185417 +</pre></div></div> +<p>After training some steps (depends on the setting) or the job is finished, SINGA will <a href="checkpoint.html">checkpoint</a> the model parameters.</p></div> +<div class="section"> +<h3><a name="Training_on_GPU"></a>Training on GPU</h3> +<p>Since version 0.2, we can train CNN models on GPU using cuDNN. Please refer to the <a href="gpu.html">GPU page</a> for details on compiling SINGA with GPU and cuDNN. The configuration file is similar to that for CPU training, except that the cuDNN layers are used and the GPU device is configured.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">./bin/singa-run.sh -conf examples/cifar10/cudnn.conf +</pre></div></div></div> +<div class="section"> +<h3><a name="Training_using_Python_script"></a>Training using Python script</h3> +<p>The python helpers coming with SINGA 0.2 make it easy to configure a training job. For example the <i>job.conf</i> is replaced with a simple python script <i>mnist_mlp.py</i> which has about 30 lines of code following the <a class="externalLink" href="http://keras.io/";>Keras API</a>.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"> # on CPU +./bin/singa-run.sh -exec tool/python/examples/cifar10_cnn.py + # on GPU +./bin/singa-run.sh -exec tool/python/examples/cifar10_cnn_cudnn.py +</pre></div></div></div></div> +<div class="section"> +<h2><a name="Details"></a>Details</h2> +<p>To train a model in SINGA, you need to prepare the datasets, and a job configuration which specifies the neural net structure, training algorithm (BP or CD), SGD update algorithm (e.g. Adagrad), number of training/test steps, etc.</p> +<div class="section"> +<h3><a name="Data_preparation"></a>Data preparation</h3> +<p>Before using SINGA, you need to write a program to convert the dataset into a format that SINGA can read. Please refer to the <a href="data.html#example---cifar-dataset">Data Preparation</a> to get details about preparing this CIFAR10 dataset.</p></div> +<div class="section"> +<h3><a name="Neural_net"></a>Neural net</h3> +<p>Figure 1 shows the net structure of the CNN model we used in this example, which is set following <a class="externalLink" href="https://code.google.com/p/cuda-convnet/source/browse/trunk/example-layers/layers-18pct.cfg.";>Alex</a> The dashed circle represents one feature transformation stage, which generally has four layers as shown in the figure. Sometimes the rectifier layer and normalization layer are omitted or swapped in one stage. For this example, there are 3 such stages.</p> +<p>Next we follow the guide in <a href="neural-net.html">neural net page</a> and <a href="layer.html">layer page</a> to write the neural net configuration.</p> + +<div style="text-align: center"> +<img src="../images/example-cnn.png" style="width: 200px" alt="" /> <br /> +<b>Figure 1 - Net structure of the CNN example.</b></img> +</div> + +<ul> + +<li> +<p>We configure an input layer to read the training/testing records from a disk file.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">layer{ + name: "data" + type: kRecordInput + store_conf { + backend: "kvfile" + path: "examples/cifar10/train_data.bin" + mean_file: "examples/cifar10/image_mean.bin" + batchsize: 64 + random_skip: 5000 + shape: 3 + shape: 32 + shape: 32 + } + exclude: kTest # exclude this layer for the testing net +} +layer{ + name: "data" + type: kRecordInput + store_conf { + backend: "kvfile" + path: "examples/cifar10/test_data.bin" + mean_file: "examples/cifar10/image_mean.bin" + batchsize: 100 + shape: 3 + shape: 32 + shape: 32 + } + exclude: kTrain # exclude this layer for the training net +} +</pre></div></div></li> +</ul> + +<ul> + +<li> +<p>We configure layers for the feature transformation as follows (all layers are built-in layers in SINGA; hyper-parameters of these layers are set according to <a class="externalLink" href="https://code.google.com/p/cuda-convnet/source/browse/trunk/example-layers/layers-18pct.cfg";>Alex’s setting</a>).</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">layer { + name: "conv1" + type: kConvolution + srclayers: "data" + convolution_conf {... } + ... +} +layer { + name: "pool1" + type: kPooling + srclayers: "conv1" + pooling_conf {... } +} +layer { + name: "relu1" + type: kReLU + srclayers:"pool1" +} +layer { + name: "norm1" + type: kLRN + lrn_conf {... } + srclayers:"relu1" +} +</pre></div></div></li> +</ul> +<p>The configurations for another 2 stages are omitted here.</p> + +<ul> + +<li> +<p>There is an <a href="layer.html#innerproductlayer">inner product layer</a> after the 3 transformation stages, which is configured with 10 output units, i.e., the number of total labels. The weight matrix Param is configured with a large weight decay scale to reduce the over-fitting.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">layer { + name: "ip1" + type: kInnerProduct + srclayers:"pool3" + innerproduct_conf { + num_output: 10 + } + param { + name: "w4" + wd_scale:250 + ... + } + param { + name: "b4" + ... + } +} +</pre></div></div></li> + +<li> +<p>The last layer is a <a href="layer.html#softmaxloss">Softmax loss layer</a></p> + +<div class="source"> +<div class="source"><pre class="prettyprint">layer{ + name: "loss" + type: kSoftmaxLoss + softmaxloss_conf{ topk:1 } + srclayers:"ip1" + srclayers: "data" +} +</pre></div></div></li> +</ul></div> +<div class="section"> +<h3><a name="Updater"></a>Updater</h3> +<p>The <a href="updater.html#updater">normal SGD updater</a> is selected. The learning rate is changed like going down stairs, and is configured using the <a href="updater.html#kfixedstep">kFixedStep</a> type.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"> updater{ + type: kSGD + weight_decay:0.004 + learning_rate { + type: kFixedStep + fixedstep_conf:{ + step:0 # lr for step 0-60000 is 0.001 + step:60000 # lr for step 60000-65000 is 0.0001 + step:65000 # lr for step 650000- is 0.00001 + step_lr:0.001 + step_lr:0.0001 + step_lr:0.00001 + } + } + } +</pre></div></div></div> +<div class="section"> +<h3><a name="TrainOneBatch_algorithm"></a>TrainOneBatch algorithm</h3> +<p>The CNN model is a feed forward model, thus should be configured to use the <a href="train-one-batch.html#back-propagation">Back-propagation algorithm</a>.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">train_one_batch { + alg: kBP +} +</pre></div></div></div> +<div class="section"> +<h3><a name="Cluster_setting"></a>Cluster setting</h3> +<p>The following configuration set a single worker and server for training. <a href="frameworks.html">Training frameworks</a> page introduces configurations of a couple of distributed training frameworks.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">cluster { + nworker_groups: 1 + nserver_groups: 1 +} +</pre></div></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html> Added: websites/staging/singa/trunk/content/v0.3.0/zh/data.html ============================================================================== --- websites/staging/singa/trunk/content/v0.3.0/zh/data.html (added) +++ websites/staging/singa/trunk/content/v0.3.0/zh/data.html Wed Apr 20 05:12:03 2016 @@ -0,0 +1,426 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-04-20 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160420" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache SINGA – Data Preparation</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; type="text/javascript"></script> + + +<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script> + + +<meta content="Apache SINGA" name="author"/> + + +<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/> + + </head> + <body class="topBarEnabled"> + + + + + + + <a href="https://github.com/apache/incubator-singa";> + <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" + src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"; + alt="Fork me on GitHub"> + </a> + + + + + + <div id="topbar" class="navbar navbar-fixed-top navbar-inverse"> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../../docs/overview.html" title="Introduction">Introduction</a> +</li> + + <li> <a href="../../docs/quick-start.html" title="Quick Start">Quick Start</a> +</li> + + <li> <a href="../../downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"> + <a href="../../docs/index.html" title="v0.3.0">v0.3.0</a> + <ul class="dropdown-menu"> + <li> <a href="../../docs/index.html" title="English">English</a> +</li> + <li> <a href="../../docs/zh/index.html" title="ä¸æ">ä¸æ</a> +</li> + <li> <a href="../../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a> +</li> + <li> <a href="../../docs/kr/index.html" title="íêµì´">íêµì´</a> +</li> + </ul> + </li> + + <li> <a href="../../v0.2.0/index.html" title="v0.2.0">v0.2.0</a> +</li> + + <li> <a href="../../v0.1.0/index.html" title="v0.1.0">v0.1.0</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../../develop/schedule.html" title="Schedule">Schedule</a> +</li> + + <li class="dropdown-submenu"> + <a href="../../develop/how-contribute.html" title="How to Contribute">How to Contribute</a> + <ul class="dropdown-menu"> + <li> <a href="../../develop/contribute-code.html" title="Code">Code</a> +</li> + <li> <a href="../../develop/contribute-docs.html" title="Documentation">Documentation</a> +</li> + </ul> + </li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="../../community/source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="../../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="../../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="../../community/team-list.html" title="SINGA Team">SINGA Team</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; title="NUS Site">NUS Site</a> +</li> + </ul> + </li> + </ul> + + + + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="../../index.html" id="bannerLeft" title="Apache SINGA"> + <img src="../../images/singa-logo.png" alt="Apache SINGA"/> + </a> + </div> + <div class="pull-right"> <div id="bannerRight"> + <img src="../../images/singa-title.png" alt="Apache SINGA"/> + </div> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="../../index.html" title="Apache SINGA"> + Apache SINGA</a> + <span class="divider">/</span> + </li> + <li class="active ">Data Preparation</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache SINGA</li> + + <li> + + <a href="../../docs/overview.html" title="Introduction"> + <span class="none"></span> + Introduction</a> + </li> + + <li> + + <a href="../../docs/quick-start.html" title="Quick Start"> + <span class="none"></span> + Quick Start</a> + </li> + + <li> + + <a href="../../downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentaion</li> + + <li> + + <a href="../../docs/index.html" title="v0.3.0"> + <span class="icon-chevron-right"></span> + v0.3.0</a> + </li> + + <li> + + <a href="../../v0.2.0/index.html" title="v0.2.0"> + <span class="none"></span> + v0.2.0</a> + </li> + + <li> + + <a href="../../v0.1.0/index.html" title="v0.1.0"> + <span class="none"></span> + v0.1.0</a> + </li> + <li class="nav-header">Development</li> + + <li> + + <a href="../../develop/schedule.html" title="Schedule"> + <span class="none"></span> + Schedule</a> + </li> + + <li> + + <a href="../../develop/how-contribute.html" title="How to Contribute"> + <span class="icon-chevron-right"></span> + How to Contribute</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="../../community/source-repository.html" title="Source Repository"> + <span class="none"></span> + Source Repository</a> + </li> + + <li> + + <a href="../../community/mail-lists.html" title="Mailing Lists"> + <span class="none"></span> + Mailing Lists</a> + </li> + + <li> + + <a href="../../community/issue-tracking.html" title="Issue Tracking"> + <span class="none"></span> + Issue Tracking</a> + </li> + + <li> + + <a href="../../community/team-list.html" title="SINGA Team"> + <span class="none"></span> + SINGA Team</a> + </li> + <li class="nav-header">External Links</li> + + <li> + + <a href="http://www.apache.org/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/"; class="externalLink" title="NUS Site"> + <span class="none"></span> + NUS Site</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://incubator.apache.org"; title="apache-incubator" class="builtBy"> + <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png"; /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <h1>Data Preparation</h1> +<hr /> +<p>SINGA uses input layers to load data. Users can store their data in any format (e.g., CSV or binary) and at any places (e.g., disk file or HDFS) as long as there are corresponding input layers that can read the data records and parse them.</p> +<p>To make it easy for users, SINGA provides a [StoreInputLayer] to read data in the format of (string:key, string:value) tuples from a couple of sources. These sources are abstracted using a <a href="">Store</a> class which is a simple version of the DB abstraction in Caffe. The base Store class provides the following operations for reading and writing tuples,</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">Open(string path, Mode mode); // open the store for kRead or kCreate or kAppend +Close(); + +Read(string* key, string* val); // read a tuple; return false if fail +Write(string key, string val); // write a tuple +Flush(); +</pre></div></div> +<p>Currently, two implementations are provided, namely</p> + +<ol style="list-style-type: decimal"> + +<li> +<p>[KVFileStore] for storing tuples in <a href="">KVFile</a> (a binary file). The <i>create_data.cc</i> files in <i>examples/cifar10</i> and <i>examples/mnist</i> provide examples of storing records using KVFileStore.</p></li> + +<li> +<p>[TextFileStore] for storing tuples in plain text file (one line per tuple).</p></li> +</ol> +<p>The (key, value) tuple are parsed by subclasses of StoreInputLayer depending on the format of the tuple,</p> + +<ul> + +<li> +<p>[ProtoRecordInputLayer] parses the value field from one tuple into a [SingleLabelImageRecord], which is generated by Google Protobuf according to [common.proto]. It can be used to store features for images (e.g., using the pixel field) or other objects (using the data field). The key field is not used.</p></li> + +<li> +<p>[CSVRecordInputLayer] parses one tuple as a CSV line (separated by comma).</p></li> +</ul> +<div class="section"> +<h2><a name="Using_built-in_record_format"></a>Using built-in record format</h2> +<p>SingleLabelImageRecord is a built-in record in SINGA for storing image features. It is used in the cifar10 and mnist examples.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">message SingleLabelImageRecord { + repeated int32 shape = 1; // it obtains 3 (rgb channels), 32 (row), 32 (col) + optional int32 label = 2; // label + optional bytes pixel = 3; // pixels + repeated float data = 4 [packed = true]; // it is used for normalization +</pre></div></div> +<p>}</p> +<p>The data preparation instructions for the <a class="externalLink" href="http://www.cs.toronto.edu/~kriz/cifar.html";>CIFAR-10 image dataset</a> will be elaborated here. This dataset consists of 60,000 32x32 color images in 10 classes, with 6,000 images per class. There are 50,000 training images and 10,000 test images. Each image has a single label. This dataset is stored in binary files with specific format. SINGA comes with the <a class="externalLink" href="https://github.com/apache/incubator-singa/blob/master/examples/cifar10/create_data.cc";>create_data.cc</a> to convert images in the binary files into <tt>SingleLabelImageRecord</tt>s and insert them into training and test stores.</p> + +<ol style="list-style-type: decimal"> + +<li> +<p>Download raw data. The following command will download the dataset into <i>cifar-10-batches-bin</i> folder.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint"># in SINGA_ROOT/examples/cifar10 +$ cp Makefile.example Makefile // an example makefile is provided +$ make download +</pre></div></div></li> + +<li> +<p>Fill one record for each image, and insert it to store.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">KVFileStore store; +store.Open(output_file_path, singa::io::kCreate); + +singa::SingleLabelImageRecord image; +for (int image_id = 0; image_id < 50000; image_id ++) { + // fill the record with image feature and label from downloaded binay files + string str; + image.SerializeToString(&str); + store.Write(to_string(image_id), str); +} +store.Flush(); +store.Close(); +</pre></div></div> +<p>The data store for testing data is created similarly. In addition, it computes average values (not shown here) of image pixels and insert the mean values into a SingleLabelImageRecord, which is then written into a another store.</p></li> + +<li> +<p>Compile and run the program. SINGA provides an example Makefile that contains instructions for compiling the source code and linking it with <i>libsinga.so</i>. Users just execute the following command.</p> + +<div class="source"> +<div class="source"><pre class="prettyprint">$ make create +</pre></div></div></li> +</ol></div> +<div class="section"> +<h2><a name="using_user-defined_record_format"></a>using user-defined record format</h2> +<p>If users cannot use the SingleLabelImageRecord or CSV record for their data. They can define their own record format e.g., using Google Protobuf. A record can be written into a data store as long as it can be converted into byte string. Correspondingly, subclasses of StoreInputLayer are required to parse user-defined records.</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + + + </div> + </footer> + </body> +</html>
![https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"](https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"){kind=link}
![http://incubator.apache.org/images/egg-logo.png"](http://incubator.apache.org/images/egg-logo.png"){kind=link}