Added: incubator/toree/site/toree-site/site/documentation/old/quick-start/comm-api.html URL: http://svn.apache.org/viewvc/incubator/toree/site/toree-site/site/documentation/old/quick-start/comm-api.html?rev=1748725&view=auto ============================================================================== --- incubator/toree/site/toree-site/site/documentation/old/quick-start/comm-api.html (added) +++ incubator/toree/site/toree-site/site/documentation/old/quick-start/comm-api.html Thu Jun 16 13:45:37 2016 @@ -0,0 +1,543 @@ + +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <title>Comm API</title> + + <meta name="author" content=""> + + <!-- Enable responsive viewport --> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <!-- Le HTML5 shim, for IE6-8 support of HTML elements --> + <!--[if lt IE 9]> + <script src="http://html5shim.googlecode.com/svn/trunk/html5.js";></script> + <![endif]--> + + <!-- Le styles --> + <link href="/assets/themes/apache/font-awesome-4.5.0/css/font-awesome.min.css"> + <link href="/assets/themes/apache/bootstrap/css/bootstrap.css" rel="stylesheet"> + + <!-- NOTE: Provides syntax highlighting to pygments classes, but looks weird --> + <!-- <link href="/assets/themes/apache/css/zenburn.css" rel="stylesheet"> --> + + <!-- Le fav and touch icons --> + <!-- Update these with your own images + <link rel="shortcut icon" href="images/favicon.ico"> + <link rel="apple-touch-icon" href="images/apple-touch-icon.png"> + <link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png"> + <link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png"> + --> + + </head> + + <body data-spy="scroll" data-target="#navbar" style="position: relative;"> + + <div class="navbar navbar-inverse navbar-static-top" role="navigation"> + <div class="container"> + <div class="navbar-header"> + <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a class="navbar-brand" href="/"> + <img src="/assets/themes/apache/img/logo.png" width="50"> + Apache Toree + </a> + </div> + <nav class="navbar-collapse collapse" role="navigation"> + <ul class="nav navbar-nav navbar-right"> + <li id="documentation"> + <a href="#" data-toggle="dropdown" class="dropdown-toggle">Documentation<b class="caret"></b></a> + <ul class="dropdown-menu dropdown-left"> + + <li><a href="/documentation/user/quick-start">User</a></li> + + <li><a href="/documentation/developer/contributing-to-the-project">Developer</a></li> + + <li><a href="/documentation/references/scaladocs">References</a></li> + + </ul> + </li> + <li id="community"> + <a href="#" data-toggle="dropdown" class="dropdown-toggle">Community<b class="caret"></b></a> + <ul class="dropdown-menu dropdown-left"> + <li><a href="/community/get-involved">Get Involved</a></li> + <li><a href="/community/members">Who we are</a></li> + </ul> + </li> + <li><a href="https://github.com/apache/incubator-toree";>GitHub</a></li> + <li id="apache"> + <a href="#" data-toggle="dropdown" class="dropdown-toggle">Apache<b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="https://www.apache.org/foundation/how-it-works.html";>Apache Software Foundation</a></li> + <li><a href="https://www.apache.org/licenses/";>Apache License</a></li> + <li><a href="https://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="https://www.apache.org/foundation/thanks.html";>Thanks</a></li> + </ul> + </li> + </ul> + </nav><!--/.navbar-collapse --> + </div> + </div> + + + + + <div class="container"> + +<!--<div class="hero-unit Comm API"> + <h1> <small>Apache Project !</small></h1> +</div> +--> + +<div class="row"> + <div class="col-md-3" role="complementary"> + <nav id="navbar" class="doc-nav" data-spy="affix" data-offset-top="60" data-offset-bottom="200"> + + + + + <style> + .doc-nav h3 { + text-transform: uppercase; + font-size: 2rem; + color: black; + } + + .doc-nav a { + color: #666; + display: block; + margin-left: 0px; + font-weight: 400; + text-decoration: none; + padding-left: 0px !important; + } + + .doc-nav a.active { + color: #C15757; + } + + .doc-nav li .doc-section { + border-bottom: 1px solid #ccc; + } + .doc-nav li:last-child .doc-section { + border-bottom: 0px; + } + </style> + <ul class="nav"> + + + + + + <li> + <div class="doc-section"> + <h3>User</h3> + <ul class="nav"> + + <li> + <a class="" + href="/documentation/user/quick-start.html">Quick Start</a> + </li> + + <li> + <a class="" + href="/documentation/user/installation.html">Installation</a> + </li> + + <li> + <a class="" + href="/documentation/user/how-it-works.html">How it works</a> + </li> + + <li> + <a class="" + href="/documentation/user/using-with-jupyter-notebooks.html">Using with Jupyter Notebooks</a> + </li> + + <li> + <a class="" + href="/documentation/user/using-standalone.html">Using Standalone</a> + </li> + + <li> + <a class="" + href="/documentation/user/faq.html">FAQ</a> + </li> + + <li> + <a class="" + href="/documentation/user/advanced-topics.html">Advanced Topics</a> + </li> + + </ul> + </div> + </li> + + + + + + <li> + <div class="doc-section"> + <h3>Developer</h3> + <ul class="nav"> + + <li> + <a class="" + href="/documentation/developer/contributing-to-the-project.html">Contributing to the Project</a> + </li> + + <li> + <a class="" + href="/documentation/developer/creating-extensions.html">Creating Extensions</a> + </li> + + </ul> + </div> + </li> + + + + + + <li> + <div class="doc-section"> + <h3>References</h3> + <ul class="nav"> + + <li> + <a class="" + href="/documentation/references/scaladocs.html">ScalaDocs</a> + </li> + + </ul> + </div> + </li> + + </ul> +</nav> + + + </div> + <div class="col-md-9" role="main"> + <p>The Comm API exposed by the Spark Kernel Client and Spark Kernel serves to +provide a clean method of communication between the Spark Kernel and its +clients.</p> + +<p>The API provides the ability to create and send custom messages with the +focus on synchronizing data between a kernel and its clients, although that +use case is not enforced.</p> + +<p>Access to the Comm API is made available for the client via +<code class="highlighter-rouge"><client_instance>.comm</code> and for the kernel via <code class="highlighter-rouge">kernel.comm</code>.</p> + +<h2 id="example-of-registration-and-communication">Example of Registration and Communication</h2> + +<p>The following example demonstrates the <em>client</em> connecting to the <em>kernel</em>, +receiving a response, and then closing itâs connection.</p> + +<p>This is an example of registering an open callback on the <em>kernel</em> side:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>// Register the callback to respond to being opened from the client +kernel.comm.register("my target").addOpenHandler { + (commWriter, commId, targetName, data) => + commWriter.writeMsg(Map("response" -> "Hello World!")) +} +</code></pre> +</div> + +<p>This is the corresponding example of registering a message receiver on the +<em>client</em> and initiating the Comm connection via <em>open</em>:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>val client: SparkKernelClient = /* Created elsewhere */ + +// Register the callback to receive a message from the kernel, print it +// out, and then close the connection +client.comm.register("my target").addMsgHandler { + (commWriter, commId, data) => + println(data("response")) + commWriter.close() +} + +// Initiate the Comm connection +client.comm.open("my target") +</code></pre> +</div> + +<h2 id="comm-events">Comm Events</h2> + +<p>The Comm API provides three types of events that can be captured:</p> + +<ol> + <li> + <p>Open</p> + + <ul> + <li>Triggered when the client/kernel receives an open request for a target +that has been registered</li> + </ul> + </li> + <li> + <p>Msg</p> + + <ul> + <li>Triggered when the client/kernel receives a Comm message for an open +Comm instance</li> + </ul> + </li> + <li> + <p>Close</p> + + <ul> + <li>Triggered when the client/kernel receives a close request for an open +Comm instance</li> + </ul> + </li> +</ol> + +<h3 id="registering-callbacks">Registering Callbacks</h3> + +<p>To register callbacks that are triggered during these events, the following +function is provided:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>register(<target name>) +</code></pre> +</div> + +<p>This function, when invoked, registers the provided target on the +client/kernel, but does not add any callbacks. To add functions to be called +during events, you can chain methods onto the register function.</p> + +<h4 id="adding-open-callbacks">Adding Open Callbacks</h4> + +<p>To add an open callback, use the <code class="highlighter-rouge">addOpenHandler(<function>)</code> method:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>register(<target name>).addOpenHandler(<function>) +</code></pre> +</div> + +<p>The function is given the following four arguments:</p> + +<ul> + <li> + <p>CommWriter</p> + + <ul> + <li>The instance of the Comm-based writer that can send messages back</li> + </ul> + </li> + <li> + <p>CommId</p> + + <ul> + <li>The id associated with the new Comm instance</li> + </ul> + </li> + <li> + <p>TargetName</p> + + <ul> + <li>The name of the Comm that is created</li> + </ul> + </li> + <li> + <p>Data (<em>Optional</em>)</p> + + <ul> + <li>The map of key/value pairs representing data associated with the new +Comm instance</li> + </ul> + </li> +</ul> + +<h4 id="adding-message-callbacks">Adding Message Callbacks</h4> + +<p>To add a message callback, use the <code class="highlighter-rouge">addMsgHandler(<function>)</code> method:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>register(<target name>).addMsgHandler(<function>) +</code></pre> +</div> + +<p>The function is given the following three arguments:</p> + +<ul> + <li> + <p>CommWriter</p> + + <ul> + <li>The instance of the Comm-based writer that can send messages back</li> + </ul> + </li> + <li> + <p>CommId</p> + + <ul> + <li>The id associated with the Comm instance</li> + </ul> + </li> + <li> + <p>Data</p> + + <ul> + <li>The map of key/value pairs representing data associated with the +received message</li> + </ul> + </li> +</ul> + +<h4 id="adding-close-callbacks">Adding Close Callbacks</h4> + +<p>To add a close callback, use the <code class="highlighter-rouge">addCloseHandler(<function>)</code> method:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>register(<target name>).addCloseHandler(<function>) +</code></pre> +</div> + +<p>The function is given the following three arguments:</p> + +<ul> + <li> + <p>CommWriter</p> + + <ul> + <li>Unused as the Comm instance associated with the writer has been closed</li> + </ul> + </li> + <li> + <p>CommId</p> + + <ul> + <li>The id associated with the Comm instance that was closed</li> + </ul> + </li> + <li> + <p>Data</p> + + <ul> + <li>The map of key/value pairs representing data associated with the +received message</li> + </ul> + </li> +</ul> + +<h2 id="comm-messaging">Comm Messaging</h2> + +<p>The Comm API exposes an <em>open</em> method that initiates a new Comm instance on +both sides of the connection:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>`open(<target name>)` +</code></pre> +</div> + +<p>This returns an instance of <em>CommWriter</em> that can be used to send data via +the Comm protocol.</p> + +<p>The kernel would initiate the connection via <code class="highlighter-rouge">kernel.comm.open(<target name>)</code> +while the client would start via <code class="highlighter-rouge"><client instance>.comm.open(<target name>)</code>.</p> + +<p>As per the IPython protocol definition, the Comm instance can be opened from +either side.</p> + +<h3 id="using-the-comm-writer">Using the Comm Writer</h3> + +<p>The Comm API provides an implementation of <a href="http://docs.oracle.com/javase/7/docs/api/java/io/Writer.html";>java.io.Writer</a> that is used to +send <em>open</em>, <em>msg</em>, and <em>close</em> Comm messages to the client or kernel (client +to kernel or vice versa).</p> + +<p>The following methods are available with <em>CommWriter</em> implementations:</p> + +<ol> + <li> + <p><code class="highlighter-rouge">writeOpen(<target name> [, data])</code></p> + + <ul> + <li>Sends an open request with the given target name and optional map of data</li> + </ul> + </li> + <li> + <p><code class="highlighter-rouge">writeMsg(<data>)</code></p> + + <ul> + <li>Sends the map of data as a Comm message</li> + </ul> + </li> + <li> + <p><code class="highlighter-rouge">write(<character array>, <offset>, <length>)</code></p> + + <ul> + <li> + <p>Sends the character array as a Comm message (in the same form as a +<em>Writerâs</em> write(â¦) method) with the key for the data as âmessageâ</p> + + <ul> + <li> + <p>E.g. <code class="highlighter-rouge">commWriter.write(<array>, 0, <array length>)</code> translates to</p> + + <p>Data(âmessageâ: â<array>")</array></p> + </li> + </ul> + </li> + </ul> + </li> + <li> + <p><code class="highlighter-rouge">writeClose([data])</code></p> + + <ul> + <li>Sends a close request with the optional map of data</li> + </ul> + </li> + <li> + <p><code class="highlighter-rouge">close()</code></p> + + <ul> + <li>Sends a close request with no data</li> + </ul> + </li> +</ol> + + + </div> +</div> + + + + <hr> + <footer> + <!-- <p>© 2016 </p>--> + <footer class="site-footer"> + + <div class="wrapper"> + + <div class="footer-col-wrapper"> + + Apache Toree is an effort undergoing <a href="https://incubator.apache.org/index.html";>Incubation</a> at The Apache Software Foundation (ASF), sponsored by the Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. + + <div style="text-align:center"> + <br><br> + Copyright © 2015 The Apache Software Foundation + <br> + Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0";>Apache License, Version 2.0</a>. + <br> + Apache, the Apache feather logo, and the Apache Incubator project logo are trademarks of The Apache Software Foundation. + </div> + </div> + + </div> + +</footer> + + </footer> + </div> + + + <script src="/assets/themes/apache/jquery/jquery-2.1.1.min.js"></script> + <script src="/assets/themes/apache/bootstrap/js/bootstrap.min.js"></script> + + </body> +</html> +
Added: incubator/toree/site/toree-site/site/documentation/old/quick-start/current-magics.html URL: http://svn.apache.org/viewvc/incubator/toree/site/toree-site/site/documentation/old/quick-start/current-magics.html?rev=1748725&view=auto ============================================================================== --- incubator/toree/site/toree-site/site/documentation/old/quick-start/current-magics.html (added) +++ incubator/toree/site/toree-site/site/documentation/old/quick-start/current-magics.html Thu Jun 16 13:45:37 2016 @@ -0,0 +1,435 @@ + +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <title>Current Magics</title> + + <meta name="author" content=""> + + <!-- Enable responsive viewport --> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <!-- Le HTML5 shim, for IE6-8 support of HTML elements --> + <!--[if lt IE 9]> + <script src="http://html5shim.googlecode.com/svn/trunk/html5.js";></script> + <![endif]--> + + <!-- Le styles --> + <link href="/assets/themes/apache/font-awesome-4.5.0/css/font-awesome.min.css"> + <link href="/assets/themes/apache/bootstrap/css/bootstrap.css" rel="stylesheet"> + + <!-- NOTE: Provides syntax highlighting to pygments classes, but looks weird --> + <!-- <link href="/assets/themes/apache/css/zenburn.css" rel="stylesheet"> --> + + <!-- Le fav and touch icons --> + <!-- Update these with your own images + <link rel="shortcut icon" href="images/favicon.ico"> + <link rel="apple-touch-icon" href="images/apple-touch-icon.png"> + <link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png"> + <link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png"> + --> + + </head> + + <body data-spy="scroll" data-target="#navbar" style="position: relative;"> + + <div class="navbar navbar-inverse navbar-static-top" role="navigation"> + <div class="container"> + <div class="navbar-header"> + <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a class="navbar-brand" href="/"> + <img src="/assets/themes/apache/img/logo.png" width="50"> + Apache Toree + </a> + </div> + <nav class="navbar-collapse collapse" role="navigation"> + <ul class="nav navbar-nav navbar-right"> + <li id="documentation"> + <a href="#" data-toggle="dropdown" class="dropdown-toggle">Documentation<b class="caret"></b></a> + <ul class="dropdown-menu dropdown-left"> + + <li><a href="/documentation/user/quick-start">User</a></li> + + <li><a href="/documentation/developer/contributing-to-the-project">Developer</a></li> + + <li><a href="/documentation/references/scaladocs">References</a></li> + + </ul> + </li> + <li id="community"> + <a href="#" data-toggle="dropdown" class="dropdown-toggle">Community<b class="caret"></b></a> + <ul class="dropdown-menu dropdown-left"> + <li><a href="/community/get-involved">Get Involved</a></li> + <li><a href="/community/members">Who we are</a></li> + </ul> + </li> + <li><a href="https://github.com/apache/incubator-toree";>GitHub</a></li> + <li id="apache"> + <a href="#" data-toggle="dropdown" class="dropdown-toggle">Apache<b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="https://www.apache.org/foundation/how-it-works.html";>Apache Software Foundation</a></li> + <li><a href="https://www.apache.org/licenses/";>Apache License</a></li> + <li><a href="https://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="https://www.apache.org/foundation/thanks.html";>Thanks</a></li> + </ul> + </li> + </ul> + </nav><!--/.navbar-collapse --> + </div> + </div> + + + + + <div class="container"> + +<!--<div class="hero-unit Current Magics"> + <h1> <small>Apache Project !</small></h1> +</div> +--> + +<div class="row"> + <div class="col-md-3" role="complementary"> + <nav id="navbar" class="doc-nav" data-spy="affix" data-offset-top="60" data-offset-bottom="200"> + + + + + <style> + .doc-nav h3 { + text-transform: uppercase; + font-size: 2rem; + color: black; + } + + .doc-nav a { + color: #666; + display: block; + margin-left: 0px; + font-weight: 400; + text-decoration: none; + padding-left: 0px !important; + } + + .doc-nav a.active { + color: #C15757; + } + + .doc-nav li .doc-section { + border-bottom: 1px solid #ccc; + } + .doc-nav li:last-child .doc-section { + border-bottom: 0px; + } + </style> + <ul class="nav"> + + + + + + <li> + <div class="doc-section"> + <h3>User</h3> + <ul class="nav"> + + <li> + <a class="" + href="/documentation/user/quick-start.html">Quick Start</a> + </li> + + <li> + <a class="" + href="/documentation/user/installation.html">Installation</a> + </li> + + <li> + <a class="" + href="/documentation/user/how-it-works.html">How it works</a> + </li> + + <li> + <a class="" + href="/documentation/user/using-with-jupyter-notebooks.html">Using with Jupyter Notebooks</a> + </li> + + <li> + <a class="" + href="/documentation/user/using-standalone.html">Using Standalone</a> + </li> + + <li> + <a class="" + href="/documentation/user/faq.html">FAQ</a> + </li> + + <li> + <a class="" + href="/documentation/user/advanced-topics.html">Advanced Topics</a> + </li> + + </ul> + </div> + </li> + + + + + + <li> + <div class="doc-section"> + <h3>Developer</h3> + <ul class="nav"> + + <li> + <a class="" + href="/documentation/developer/contributing-to-the-project.html">Contributing to the Project</a> + </li> + + <li> + <a class="" + href="/documentation/developer/creating-extensions.html">Creating Extensions</a> + </li> + + </ul> + </div> + </li> + + + + + + <li> + <div class="doc-section"> + <h3>References</h3> + <ul class="nav"> + + <li> + <a class="" + href="/documentation/references/scaladocs.html">ScalaDocs</a> + </li> + + </ul> + </div> + </li> + + </ul> +</nav> + + + </div> + <div class="col-md-9" role="main"> + <p>Magic names are case insensitive, so <em>AddJar</em> is the same as <em>addjar</em>.</p> + +<h2 id="line-magics">Line Magics</h2> + +<ul> + <li> + <p><strong>AddDeps</strong> _<my.company> <artifact-id> <version> [--transitive]_</version></artifact-id></my.company></p> + + <ul> + <li> + <p>Adds the specified dependencies from Maven Central to the Spark Kernel +and Spark cluster</p> + </li> + <li> + <p>Requires the company name, artifact id, and version of the dependency</p> + </li> + <li> + <p>Including <em>âtransitive</em> will include all needed dependencies</p> + </li> + <li> + <p>Examples:</p> + <ul> + <li> + <p>Adding a single library with all of its dependencies + <code class="highlighter-rouge">scala + %AddDeps org.apache.spark spark-streaming-kafka_2.10 1.1.0 --transitive + </code></p> + </li> + <li> + <p>Using the programmatic API + <code class="highlighter-rouge">scala + kernel.magics.addDeps("org.apache.spark spark-streaming-kafka_2.10 1.1.0 --transitive") + </code></p> + </li> + </ul> + </li> + </ul> + </li> + <li> + <p><strong>AddJar</strong> _<jar-path> [-f]_</jar-path></p> + + <ul> + <li> + <p>Adds the specified jars to the Spark Kernel and Spark cluster</p> + </li> + <li> + <p>Requires the path to the jar, which can either be a local path or remote +jar hosted via HTTP</p> + </li> + <li> + <p>Including <em>-f</em> will ignore any cached jars and redownload remote +jars</p> + </li> + <li> + <p>Examples:</p> + + <ul> + <li> + <p>Adding a single jar from HTTP and forcing redownload if cached + <code class="highlighter-rouge">scala + %AddJar http://example.com/some_lib.jar -f + </code></p> + </li> + <li> + <p>Adding a single jar from the file system relative to the kernel + <code class="highlighter-rouge">scala + %AddJar file:/path/to/some/lib.jar + </code></p> + </li> + <li> + <p>Using the programmatic API + <code class="highlighter-rouge">scala + kernel.magics.addJar("http://example.com/some_lib.jar -f") + </code></p> + </li> + </ul> + </li> + </ul> + </li> + <li> + <p><strong>LSMagic</strong></p> + + <ul> + <li> + <p>Lists all available magics in the kernel</p> + </li> + <li> + <p>Examples:</p> + + <ul> + <li> + <p>Percent syntax + <code class="highlighter-rouge">scala + %LSMagic + </code></p> + </li> + <li> + <p>Using the programmatic API + <code class="highlighter-rouge">scala + kernel.magics.lsMagic() + </code></p> + </li> + </ul> + </li> + </ul> + </li> +</ul> + +<h2 id="cell-magics">Cell Magics</h2> + +<ul> + <li> + <p><strong>Html</strong> _<html-code>_</html-code></p> + + <ul> + <li> + <p>Returns the given code with a HTML MIME-type.</p> + </li> + <li> + <p>Examples:</p> + + <ul> + <li>Percent syntax + ```scala + %%Html + <h1>Hello</h1> + <p>World</p> + <p>```</p> + </li> + <li>Using the programmatic API + <code class="highlighter-rouge">scala + kernel.magics.html("<h1>Hello</h1><p>World</p>") + </code></li> + </ul> + </li> + </ul> + </li> + <li> + <p><strong>JavaScript</strong> _<javascript-code>_</javascript-code></p> + + <ul> + <li> + <p>Returns the given code with an <code class="highlighter-rouge">application/javascript</code> MIME-type.</p> + </li> + <li> + <p>Examples:</p> + + <ul> + <li> + <p>Percent syntax + <code class="highlighter-rouge">scala + %%JavaScript + var x = 3 + var y = x + 2 + </code></p> + </li> + <li> + <p>Using the programmatic API + <code class="highlighter-rouge">scala + kernel.magics.javascript("var x = 3; var y = x + 2") + </code></p> + </li> + </ul> + </li> + </ul> + </li> +</ul> + + </div> +</div> + + + + <hr> + <footer> + <!-- <p>© 2016 </p>--> + <footer class="site-footer"> + + <div class="wrapper"> + + <div class="footer-col-wrapper"> + + Apache Toree is an effort undergoing <a href="https://incubator.apache.org/index.html";>Incubation</a> at The Apache Software Foundation (ASF), sponsored by the Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. + + <div style="text-align:center"> + <br><br> + Copyright © 2015 The Apache Software Foundation + <br> + Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0";>Apache License, Version 2.0</a>. + <br> + Apache, the Apache feather logo, and the Apache Incubator project logo are trademarks of The Apache Software Foundation. + </div> + </div> + + </div> + +</footer> + + </footer> + </div> + + + <script src="/assets/themes/apache/jquery/jquery-2.1.1.min.js"></script> + <script src="/assets/themes/apache/bootstrap/js/bootstrap.min.js"></script> + + </body> +</html> + Added: incubator/toree/site/toree-site/site/documentation/old/quick-start/developing-magics.html URL: http://svn.apache.org/viewvc/incubator/toree/site/toree-site/site/documentation/old/quick-start/developing-magics.html?rev=1748725&view=auto ============================================================================== --- incubator/toree/site/toree-site/site/documentation/old/quick-start/developing-magics.html (added) +++ incubator/toree/site/toree-site/site/documentation/old/quick-start/developing-magics.html Thu Jun 16 13:45:37 2016 @@ -0,0 +1,467 @@ + +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <title>Developing Magics</title> + + <meta name="author" content=""> + + <!-- Enable responsive viewport --> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <!-- Le HTML5 shim, for IE6-8 support of HTML elements --> + <!--[if lt IE 9]> + <script src="http://html5shim.googlecode.com/svn/trunk/html5.js";></script> + <![endif]--> + + <!-- Le styles --> + <link href="/assets/themes/apache/font-awesome-4.5.0/css/font-awesome.min.css"> + <link href="/assets/themes/apache/bootstrap/css/bootstrap.css" rel="stylesheet"> + + <!-- NOTE: Provides syntax highlighting to pygments classes, but looks weird --> + <!-- <link href="/assets/themes/apache/css/zenburn.css" rel="stylesheet"> --> + + <!-- Le fav and touch icons --> + <!-- Update these with your own images + <link rel="shortcut icon" href="images/favicon.ico"> + <link rel="apple-touch-icon" href="images/apple-touch-icon.png"> + <link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png"> + <link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png"> + --> + + </head> + + <body data-spy="scroll" data-target="#navbar" style="position: relative;"> + + <div class="navbar navbar-inverse navbar-static-top" role="navigation"> + <div class="container"> + <div class="navbar-header"> + <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a class="navbar-brand" href="/"> + <img src="/assets/themes/apache/img/logo.png" width="50"> + Apache Toree + </a> + </div> + <nav class="navbar-collapse collapse" role="navigation"> + <ul class="nav navbar-nav navbar-right"> + <li id="documentation"> + <a href="#" data-toggle="dropdown" class="dropdown-toggle">Documentation<b class="caret"></b></a> + <ul class="dropdown-menu dropdown-left"> + + <li><a href="/documentation/user/quick-start">User</a></li> + + <li><a href="/documentation/developer/contributing-to-the-project">Developer</a></li> + + <li><a href="/documentation/references/scaladocs">References</a></li> + + </ul> + </li> + <li id="community"> + <a href="#" data-toggle="dropdown" class="dropdown-toggle">Community<b class="caret"></b></a> + <ul class="dropdown-menu dropdown-left"> + <li><a href="/community/get-involved">Get Involved</a></li> + <li><a href="/community/members">Who we are</a></li> + </ul> + </li> + <li><a href="https://github.com/apache/incubator-toree";>GitHub</a></li> + <li id="apache"> + <a href="#" data-toggle="dropdown" class="dropdown-toggle">Apache<b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="https://www.apache.org/foundation/how-it-works.html";>Apache Software Foundation</a></li> + <li><a href="https://www.apache.org/licenses/";>Apache License</a></li> + <li><a href="https://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="https://www.apache.org/foundation/thanks.html";>Thanks</a></li> + </ul> + </li> + </ul> + </nav><!--/.navbar-collapse --> + </div> + </div> + + + + + <div class="container"> + +<!--<div class="hero-unit Developing Magics"> + <h1> <small>Apache Project !</small></h1> +</div> +--> + +<div class="row"> + <div class="col-md-3" role="complementary"> + <nav id="navbar" class="doc-nav" data-spy="affix" data-offset-top="60" data-offset-bottom="200"> + + + + + <style> + .doc-nav h3 { + text-transform: uppercase; + font-size: 2rem; + color: black; + } + + .doc-nav a { + color: #666; + display: block; + margin-left: 0px; + font-weight: 400; + text-decoration: none; + padding-left: 0px !important; + } + + .doc-nav a.active { + color: #C15757; + } + + .doc-nav li .doc-section { + border-bottom: 1px solid #ccc; + } + .doc-nav li:last-child .doc-section { + border-bottom: 0px; + } + </style> + <ul class="nav"> + + + + + + <li> + <div class="doc-section"> + <h3>User</h3> + <ul class="nav"> + + <li> + <a class="" + href="/documentation/user/quick-start.html">Quick Start</a> + </li> + + <li> + <a class="" + href="/documentation/user/installation.html">Installation</a> + </li> + + <li> + <a class="" + href="/documentation/user/how-it-works.html">How it works</a> + </li> + + <li> + <a class="" + href="/documentation/user/using-with-jupyter-notebooks.html">Using with Jupyter Notebooks</a> + </li> + + <li> + <a class="" + href="/documentation/user/using-standalone.html">Using Standalone</a> + </li> + + <li> + <a class="" + href="/documentation/user/faq.html">FAQ</a> + </li> + + <li> + <a class="" + href="/documentation/user/advanced-topics.html">Advanced Topics</a> + </li> + + </ul> + </div> + </li> + + + + + + <li> + <div class="doc-section"> + <h3>Developer</h3> + <ul class="nav"> + + <li> + <a class="" + href="/documentation/developer/contributing-to-the-project.html">Contributing to the Project</a> + </li> + + <li> + <a class="" + href="/documentation/developer/creating-extensions.html">Creating Extensions</a> + </li> + + </ul> + </div> + </li> + + + + + + <li> + <div class="doc-section"> + <h3>References</h3> + <ul class="nav"> + + <li> + <a class="" + href="/documentation/references/scaladocs.html">ScalaDocs</a> + </li> + + </ul> + </div> + </li> + + </ul> +</nav> + + + </div> + <div class="col-md-9" role="main"> + <p>The Spark Kernel provides a pluggable interface for magics that allows +developers to write their own magics. This guide will focus on the technical details of implementing your own magics; for an introduction and conceptual overview of magics, see <a href="https://github.com/ibm-et/spark-kernel/wiki/Overview-of-Magics-for-the-Spark-Kernel";>Overview of Magics for the Spark Kernel</a>.</p> + +<p>In this guide weâll look at the dependencies required to develop a magic, walk through creating a line magic and a cell magic, and discuss some useful magic features.</p> + +<h3 id="dependencies">Dependencies</h3> + +<p>In order to write a magic, you need to add the <em>kernel-api</em> and <em>protocol</em> +modules of the Spark Kernel to your project.</p> + +<p>In <em>sbt</em>, you can add the following lines:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>libraryDependencies ++= Seq( + "com.ibm.spark" %% "kernel-api" % "0.1.1-SNAPSHOT", + "com.ibm.spark" %% "protocol" % "0.1.1-SNAPSHOT" +) +</code></pre> +</div> + +<p>As the modules are not hosted on any repository, you will also need to build +and publish them locally. From the root of the Spark Kernel, you can execute +the following to compile and make available the Spark Kernel modules:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>sbt compile && sbt publishLocal +</code></pre> +</div> + +<h2 id="developing-magics">Developing Magics</h2> + +<p>A magic is implemented by extending either the <code class="highlighter-rouge">LineMagic</code> or <code class="highlighter-rouge">CellMagic</code> trait provided by the Spark Kernel. Each trait consists of a single function, <code class="highlighter-rouge">execute</code>, that defines the magicâs functionality.</p> + +<h3 id="developing-a-line-magic">Developing a Line Magic</h3> + +<p>A line magic receives a line of code as input and performs an operation. To create a line magic, we extend the <code class="highlighter-rouge">LineMagic</code> trait, and override its <code class="highlighter-rouge">execute</code> method:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">class</span> <span class="nc">HelloLineMagic</span> <span class="k">extends</span> <span class="nc">LineMagic</span> <span class="o">{</span> + <span class="k">override</span> <span class="k">def</span> <span class="n">execute</span><span class="o">(</span><span class="n">code</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">LineMagicOutput</span> <span class="o">=</span> <span class="o">{</span> + <span class="c1">// do stuff +</span> <span class="o">}</span> + <span class="o">}</span> +</code></pre> +</div> + +<p>Note that line magics arenât meant to return values; their return type <code class="highlighter-rouge">LineMagicOutput</code> is equivalent to <code class="highlighter-rouge">Unit</code>.</p> + +<h4 id="using-the-magic">Using the Magic</h4> +<p>Now we can use our magic with either: +<code class="highlighter-rouge">scala +%helloLineMagic foo bar +</code> +or +<code class="highlighter-rouge">scala +kernel.magics.helloLineMagic("foo bar") +</code></p> + +<p>Behind the scenes, the <code class="highlighter-rouge">execute</code> method of <code class="highlighter-rouge">HelloLineMagic</code> gets called with <code class="highlighter-rouge">"foo bar"</code> as input.</p> + +<h3 id="developing-a-cell-magic">Developing a Cell Magic</h3> +<p>A cell magic receives an entire cell of code as input and returns a mapping of MIME types to data. This mapping, defined by the type <code class="highlighter-rouge">CellMagicOutput</code>, can be used to distinguish different data types produced by the magic. In an IPython setting, the <code class="highlighter-rouge">CellMagicOutput</code> mapping will influence the way a cell is rendered.</p> + +<h4 id="an-html-cell-magic">An HTML Cell Magic</h4> +<p>As a concrete example, weâll develop an <code class="highlighter-rouge">HTML</code> cell magic that causes a cell to render its contents as HTML.</p> + +<p>To create a cell magic, we extend the <code class="highlighter-rouge">CellMagic</code> trait, and override its <code class="highlighter-rouge">execute</code> method:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">class</span> <span class="nc">Html</span> <span class="k">extends</span> <span class="nc">CellMagic</span> <span class="o">{</span> + <span class="k">override</span> <span class="k">def</span> <span class="n">execute</span><span class="o">(</span><span class="n">code</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">CellMagicOutput</span> <span class="o">=</span> <span class="o">{</span> + <span class="c1">// TODO +</span> <span class="o">}</span> + <span class="o">}</span> +</code></pre> +</div> + +<p>In this case, we want to package the code that the magic receives as HTML. To do so, we return a <code class="highlighter-rouge">CellMagicOutput</code> that maps <code class="highlighter-rouge">MIMEType.TextHtml</code> to the code received:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">class</span> <span class="nc">Html</span> <span class="k">extends</span> <span class="nc">CellMagic</span> <span class="o">{</span> + <span class="k">override</span> <span class="k">def</span> <span class="n">execute</span><span class="o">(</span><span class="n">code</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">CellMagicOutput</span> <span class="o">=</span> <span class="o">{</span> + <span class="nc">CellMagicOutput</span><span class="o">(</span><span class="nc">MIMEType</span><span class="o">.</span><span class="nc">TextHtml</span> <span class="o">-></span> <span class="n">code</span><span class="o">)</span> + <span class="o">}</span> + <span class="o">}</span> +</code></pre> +</div> + +<h4 id="using-the-magic-1">Using the Magic</h4> +<p>We can use the magic with either: +```scala +%%HTML</p> +<h1>a title</h1> +<p>some more HTML</p> +<div class="highlighter-rouge"><pre class="highlight"><code>or +```scala +kernel.magics.html("<h1>a title</h1><p>some more HTML</p>") +</code></pre> +</div> + +<p>Behind the scenes, the <code class="highlighter-rouge">execute</code> method of <code class="highlighter-rouge">Html</code> gets called with <code class="highlighter-rouge">"<h1>a title</h1><p>some more HTML</p>"</code> as input.</p> + +<p>In an IPython setting, the <code class="highlighter-rouge">CellMagicOutput</code> mapping that our magic returns will cause the cell to be rendered as HTML:</p> + +<p><img src="https://raw.githubusercontent.com/wiki/ibm-et/spark-kernel/magics_html_example.png"; alt="htmlexample" /></p> + +<p>Making <code class="highlighter-rouge">Magic</code> has never been this easy!</p> + +<h3 id="argument-parsing">Argument Parsing</h3> + +<p>Sometimes, itâs useful to view the code received by the magic as a string of arguments. To help with parsing arguments out of the raw code string, you can include the <code class="highlighter-rouge">ArgumentParsingSupport</code> trait into your magic definition:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">HelloParsing</span> <span class="k">extends</span> <span class="nc">LineMagic</span> <span class="k">with</span> <span class="nc">ArgumentParsingSupport</span> <span class="o">{</span> + <span class="k">val</span> <span class="n">flag</span> <span class="k">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">accepts</span><span class="o">(</span><span class="s">"flag"</span><span class="o">,</span> <span class="s">"a boolean flag"</span><span class="o">)</span> + <span class="o">.</span><span class="n">withOptionalArg</span><span class="o">()</span> + <span class="o">.</span><span class="n">ofType</span><span class="o">(</span><span class="n">classOf</span><span class="o">[</span><span class="kt">Boolean</span><span class="o">])</span> + <span class="o">.</span><span class="n">defaultsTo</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> + + <span class="k">override</span> <span class="k">def</span> <span class="n">execute</span><span class="o">(</span><span class="n">code</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">LineMagicOutput</span> <span class="o">=</span> <span class="o">{</span> + <span class="k">val</span> <span class="n">args</span> <span class="k">=</span> <span class="n">parseArgs</span><span class="o">(</span><span class="n">code</span><span class="o">)</span> + <span class="k">if</span> <span class="o">(</span><span class="n">args</span><span class="o">(</span><span class="mi">0</span><span class="o">))</span> <span class="c1">// do something +</span> <span class="k">else</span> <span class="c1">// do something else +</span> <span class="o">}</span> +<span class="o">}</span> +</code></pre> +</div> +<p>### Adding Dependencies to Magics ###</p> + +<p>For more practical uses of magics, there are various dependencies that need to +be brought in when the magic is executed. To facilitate dependency injection, +you add the desired dependency as an additional trait to the magic.</p> + +<p>Currently, there are four available dependencies:</p> + +<ol> + <li> + <p>SparkContext - use the trait <em>IncludeSparkContext</em> to bring in the context +used by the kernel. This exposes the <em>sparkContext</em> variable into the magic.</p> + </li> + <li> + <p>Interpreter - use the trait <em>IncludeInterpreter</em> to bring in the interpreter +used by the kernel. This exposes the <em>interpreter</em> variable into the magic.</p> + </li> + <li> + <p>OutputStream - use the trait <em>IncludeOutputStream</em> to bring in an output +stream that directs output back to the client of the kernel (useful for +streaming status updates). This exposes the <em>outputStream</em> variable into +the magic.</p> + </li> + <li> + <p>DependencyDownloader - use the trait <em>IncludeDependencyDownloader</em> to bring +in the tool used to support the <em>%AddDeps</em> magic. This exposes the +<em>dependencyDownloader</em> variable into the magic.</p> + </li> +</ol> + +<p>As dependencies are injected, there is no guarantee that referencing a +dependency outside of a method will work. Instead, mark referencing code as +either <em>lazy</em> or as a helper method.</p> + +<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">class</span> <span class="nc">HelloWorld</span> <span class="k">extends</span> <span class="nc">LineMagic</span> <span class="k">with</span> <span class="nc">IncludeOutputStream</span> <span class="o">{</span> + <span class="k">val</span> <span class="n">printStream</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">PrintStream</span><span class="o">(</span><span class="n">outputStream</span><span class="o">)</span> + + <span class="c1">// ... +</span> <span class="o">}</span> +</code></pre> +</div> +<p>should become</p> + +<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">class</span> <span class="nc">HelloWorld</span> <span class="k">extends</span> <span class="nc">LineMagic</span> <span class="k">with</span> <span class="nc">IncludeOutputStream</span> <span class="o">{</span> + <span class="k">lazy</span> <span class="k">val</span> <span class="n">printStream</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">PrintStream</span><span class="o">(</span><span class="n">outputStream</span><span class="o">)</span> + + <span class="c1">// ... +</span> <span class="o">}</span> +</code></pre> +</div> + +<h3 id="adding-an-external-magic-to-the-spark-kernel">Adding an external magic to the Spark Kernel</h3> +<p>In order to use an external magic we first need a <code class="highlighter-rouge">.jar</code> containing a magic in the <code class="highlighter-rouge">com.ibm.spark.magic.builtin</code> package. Assuming we have such a <code class="highlighter-rouge">.jar</code> at location <code class="highlighter-rouge">/src/path/to/my/exampleMagic.jar</code> the <code class="highlighter-rouge">kernel.json</code> file needs to be changed to supply the path to the external magic. The command-line argument we need to add is <code class="highlighter-rouge">--magic-url</code> which takes a string:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w"> + </span><span class="nt">"display_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Spark 1.2.1 (Scala 2.10.4)"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"language_info"</span><span class="p">:</span><span class="w"> </span><span class="s2">"scala"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"argv"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> + </span><span class="s2">"/home/vagrant/local/bin/sparkkernel"</span><span class="p">,</span><span class="w"> + </span><span class="s2">"--profile"</span><span class="p">,</span><span class="w"> + </span><span class="s2">"{connection_file}"</span><span class="p">,</span><span class="w"> + </span><span class="s2">"--magic-url"</span><span class="p">,</span><span class="w"> + </span><span class="s2">"/src/path/to/my/exampleMagic.jar"</span><span class="w"> + </span><span class="p">],</span><span class="w"> + </span><span class="nt">"codemirror_mode"</span><span class="p">:</span><span class="w"> </span><span class="s2">"scala"</span><span class="w"> +</span><span class="p">}</span><span class="w"> +</span></code></pre> +</div> + +<p>Then on start-up, the kernel will load the magic from that location and it can be used: +<img src="https://raw.githubusercontent.com/wiki/ibm-et/spark-kernel/external_magic_example.png"; alt="example external magic" /> +### Examples ###</p> + +<p>For some example implementations, check out the <code class="highlighter-rouge">com.ibm.spark.magic.builtin</code> package in the <code class="highlighter-rouge">kernel</code> project folder.</p> + +<h3 id="other-notes">Other Notes</h3> + +<p>There is a limitation with the current magic implementation that will force magic invocations to be case sensitive unless defined in the package <em>com.ibm.spark.magic.builtin</em>.</p> + + </div> +</div> + + + + <hr> + <footer> + <!-- <p>© 2016 </p>--> + <footer class="site-footer"> + + <div class="wrapper"> + + <div class="footer-col-wrapper"> + + Apache Toree is an effort undergoing <a href="https://incubator.apache.org/index.html";>Incubation</a> at The Apache Software Foundation (ASF), sponsored by the Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. + + <div style="text-align:center"> + <br><br> + Copyright © 2015 The Apache Software Foundation + <br> + Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0";>Apache License, Version 2.0</a>. + <br> + Apache, the Apache feather logo, and the Apache Incubator project logo are trademarks of The Apache Software Foundation. + </div> + </div> + + </div> + +</footer> + + </footer> + </div> + + + <script src="/assets/themes/apache/jquery/jquery-2.1.1.min.js"></script> + <script src="/assets/themes/apache/bootstrap/js/bootstrap.min.js"></script> + + </body> +</html> + Added: incubator/toree/site/toree-site/site/documentation/old/quick-start/development-workflow.html URL: http://svn.apache.org/viewvc/incubator/toree/site/toree-site/site/documentation/old/quick-start/development-workflow.html?rev=1748725&view=auto ============================================================================== --- incubator/toree/site/toree-site/site/documentation/old/quick-start/development-workflow.html (added) +++ incubator/toree/site/toree-site/site/documentation/old/quick-start/development-workflow.html Thu Jun 16 13:45:37 2016 @@ -0,0 +1,392 @@ + +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <title>Development Workflow</title> + + <meta name="author" content=""> + + <!-- Enable responsive viewport --> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <!-- Le HTML5 shim, for IE6-8 support of HTML elements --> + <!--[if lt IE 9]> + <script src="http://html5shim.googlecode.com/svn/trunk/html5.js";></script> + <![endif]--> + + <!-- Le styles --> + <link href="/assets/themes/apache/font-awesome-4.5.0/css/font-awesome.min.css"> + <link href="/assets/themes/apache/bootstrap/css/bootstrap.css" rel="stylesheet"> + + <!-- NOTE: Provides syntax highlighting to pygments classes, but looks weird --> + <!-- <link href="/assets/themes/apache/css/zenburn.css" rel="stylesheet"> --> + + <!-- Le fav and touch icons --> + <!-- Update these with your own images + <link rel="shortcut icon" href="images/favicon.ico"> + <link rel="apple-touch-icon" href="images/apple-touch-icon.png"> + <link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png"> + <link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png"> + --> + + </head> + + <body data-spy="scroll" data-target="#navbar" style="position: relative;"> + + <div class="navbar navbar-inverse navbar-static-top" role="navigation"> + <div class="container"> + <div class="navbar-header"> + <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a class="navbar-brand" href="/"> + <img src="/assets/themes/apache/img/logo.png" width="50"> + Apache Toree + </a> + </div> + <nav class="navbar-collapse collapse" role="navigation"> + <ul class="nav navbar-nav navbar-right"> + <li id="documentation"> + <a href="#" data-toggle="dropdown" class="dropdown-toggle">Documentation<b class="caret"></b></a> + <ul class="dropdown-menu dropdown-left"> + + <li><a href="/documentation/user/quick-start">User</a></li> + + <li><a href="/documentation/developer/contributing-to-the-project">Developer</a></li> + + <li><a href="/documentation/references/scaladocs">References</a></li> + + </ul> + </li> + <li id="community"> + <a href="#" data-toggle="dropdown" class="dropdown-toggle">Community<b class="caret"></b></a> + <ul class="dropdown-menu dropdown-left"> + <li><a href="/community/get-involved">Get Involved</a></li> + <li><a href="/community/members">Who we are</a></li> + </ul> + </li> + <li><a href="https://github.com/apache/incubator-toree";>GitHub</a></li> + <li id="apache"> + <a href="#" data-toggle="dropdown" class="dropdown-toggle">Apache<b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="https://www.apache.org/foundation/how-it-works.html";>Apache Software Foundation</a></li> + <li><a href="https://www.apache.org/licenses/";>Apache License</a></li> + <li><a href="https://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li> + <li><a href="https://www.apache.org/foundation/thanks.html";>Thanks</a></li> + </ul> + </li> + </ul> + </nav><!--/.navbar-collapse --> + </div> + </div> + + + + + <div class="container"> + +<!--<div class="hero-unit Development Workflow"> + <h1> <small>Apache Project !</small></h1> +</div> +--> + +<div class="row"> + <div class="col-md-3" role="complementary"> + <nav id="navbar" class="doc-nav" data-spy="affix" data-offset-top="60" data-offset-bottom="200"> + + + + + <style> + .doc-nav h3 { + text-transform: uppercase; + font-size: 2rem; + color: black; + } + + .doc-nav a { + color: #666; + display: block; + margin-left: 0px; + font-weight: 400; + text-decoration: none; + padding-left: 0px !important; + } + + .doc-nav a.active { + color: #C15757; + } + + .doc-nav li .doc-section { + border-bottom: 1px solid #ccc; + } + .doc-nav li:last-child .doc-section { + border-bottom: 0px; + } + </style> + <ul class="nav"> + + + + + + <li> + <div class="doc-section"> + <h3>User</h3> + <ul class="nav"> + + <li> + <a class="" + href="/documentation/user/quick-start.html">Quick Start</a> + </li> + + <li> + <a class="" + href="/documentation/user/installation.html">Installation</a> + </li> + + <li> + <a class="" + href="/documentation/user/how-it-works.html">How it works</a> + </li> + + <li> + <a class="" + href="/documentation/user/using-with-jupyter-notebooks.html">Using with Jupyter Notebooks</a> + </li> + + <li> + <a class="" + href="/documentation/user/using-standalone.html">Using Standalone</a> + </li> + + <li> + <a class="" + href="/documentation/user/faq.html">FAQ</a> + </li> + + <li> + <a class="" + href="/documentation/user/advanced-topics.html">Advanced Topics</a> + </li> + + </ul> + </div> + </li> + + + + + + <li> + <div class="doc-section"> + <h3>Developer</h3> + <ul class="nav"> + + <li> + <a class="" + href="/documentation/developer/contributing-to-the-project.html">Contributing to the Project</a> + </li> + + <li> + <a class="" + href="/documentation/developer/creating-extensions.html">Creating Extensions</a> + </li> + + </ul> + </div> + </li> + + + + + + <li> + <div class="doc-section"> + <h3>References</h3> + <ul class="nav"> + + <li> + <a class="" + href="/documentation/references/scaladocs.html">ScalaDocs</a> + </li> + + </ul> + </div> + </li> + + </ul> +</nav> + + + </div> + <div class="col-md-9" role="main"> + <p>While it is not necessary to follow this guide for development, it is being +documented to encourage some form of standard practice for this project.</p> + +<h3 id="tooling">Tooling</h3> + +<p>Most of the developers for the Spark Kernel thus far have chosen to use +<em>IntelliJ</em> as their means of development. Because of this, a plugin for <em>sbt</em> +is included in our project to allow easy construction of an IntelliJ project +that contains all of the modules.</p> + +<p>Obviously, <em>git</em> is used as the source control for the project.</p> + +<p>Finally, we use <em>sbt</em> for our build and test runner. You can find more +information about compiling/testing in the main RAEDME.</p> + +<h3 id="building-intellij-project">Building IntelliJ Project</h3> + +<p>To build the IntelliJ project using <em>sbt</em>, you can trigger the plugin by +executing the following from the root of the Spark Kernel project:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>sbt gen-idea +</code></pre> +</div> + +<p>This should create <em>.idea/</em> and <em>.idea_modules/</em> directories.</p> + +<p>From there, you should be able to open (not import) the project using IntelliJ.</p> + +<h3 id="using-branches-for-development">Using Branches for Development</h3> + +<p>When we tackle defects or features in the Spark Kernel, we typically break the +problems up into the smallest pieces possible. Once we have something simple +like âI need the kernel to print out hello world when it starts,â we create a +branch from our development branch (in the case of this project, it is +typically master). For this example, letâs call the branch +âAddHelloWorldDuringBootâ and use it for our feature.</p> + +<p>Once development has finished, it is good practice to ensure that all tests +are still passing. To do this, run <code class="highlighter-rouge">sbt test</code> from the root of the Spark Kernel +project.</p> + +<p>If everything passes, we want to ensure that our branch is up-to-date with the +latest code in the kernel. So, move back to the development branch (master in +our case) and pull the latest changes. If there are changes, we want to rebase +our branch on top of those new changes. From the <em>AddHelloWorldDuringBoot</em> +branch, run <code class="highlighter-rouge">git rebase master</code> to bring the branch up to speed with master.</p> + +<p>The advantage of using rebase on a <em>local</em> branch is that it makes merging back +with <em>master</em> much cleaner for the maintainers. If your branch has been pushed +remotely, you want to avoid rebasing in case someone else has branched off of +your branch. Tricky stuff!</p> + +<p>After rebasing on top of master, it is a good idea to rerun the tests for your +branch to ensure that nothing has broken from the changes: <code class="highlighter-rouge">sbt test</code></p> + +<p>Finally, if the tests pass, switch back to the development branch (master) and +merge the changes: <code class="highlighter-rouge">git merge AddHelloWorldDuringBoot</code>. As a last check, +rerun the tests to ensure that the merge went well (<code class="highlighter-rouge">sbt test</code> in master). If +those tests still pass, the changes can be pushed!</p> + +<h3 id="writing-proper-unit-tests">Writing proper unit tests</h3> + +<p>The goal of our unit tests was to be isolated. This means that absolutely <em>no</em> +external logic is needed to run the tests. This includes fixtures and any +possible dependencies referenced in the code. We use <em>Mockito</em> to provide +mocking facilities for our dependencies and try our best to isolate dependency +creation.</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>class MyClass { + // Bad design + val someDependency = new SomeDependency() + + // ... +} +</code></pre> +</div> + +<p>instead move it to the constructor</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>class MyClass(someDependency: SomeDependency) { + // ... +} +</code></pre> +</div> + +<p>or use traits to mix in dependencies</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>trait MyDependency { + val someDependency = new SomeDependency() +} + +class MyClass extends MyDependency { + +} +</code></pre> +</div> + +<p>For testing, we use <em>ScalaTest</em> with the <em>FunSpec</em> to provide the basic +structure of our tests (in a BDD manner). Typically, <em>Matchers</em> from +<em>ScalaTest</em> are also included to provide a better flow.</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>class MyClassSpec extends FunSpec with Matchers { + describe("MyClass") { + describe("#someMethod") { + it("should indicate success by default") { + val myClass = new MyClass(new SomeDependency()) + val expected = true + val actual = myClass.someMethod() + + actual should be (expected) + } + } + } +} +</code></pre> +</div> + +<p>The above structure is to use a <em>describe</em> block to represent the name of the +class being tested. We nest a second layer of <em>describe</em> blocks to indicate +tests for individual public methods. Finally, <em>it</em> blocks are structured to +test single cases (such as different logical routes to be encountered).</p> + +<p>We have attempted to keep the majority of our tests clear and concise. +Typically, we avoid helper functions because they can obfuscate the tests.</p> + + </div> +</div> + + + + <hr> + <footer> + <!-- <p>© 2016 </p>--> + <footer class="site-footer"> + + <div class="wrapper"> + + <div class="footer-col-wrapper"> + + Apache Toree is an effort undergoing <a href="https://incubator.apache.org/index.html";>Incubation</a> at The Apache Software Foundation (ASF), sponsored by the Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. + + <div style="text-align:center"> + <br><br> + Copyright © 2015 The Apache Software Foundation + <br> + Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0";>Apache License, Version 2.0</a>. + <br> + Apache, the Apache feather logo, and the Apache Incubator project logo are trademarks of The Apache Software Foundation. + </div> + </div> + + </div> + +</footer> + + </footer> + </div> + + + <script src="/assets/themes/apache/jquery/jquery-2.1.1.min.js"></script> + <script src="/assets/themes/apache/bootstrap/js/bootstrap.min.js"></script> + + </body> +</html> +
![https://raw.githubusercontent.com/wiki/ibm-et/spark-kernel/magics_html_example.png"](https://raw.githubusercontent.com/wiki/ibm-et/spark-kernel/magics_html_example.png"){kind=link}
![https://raw.githubusercontent.com/wiki/ibm-et/spark-kernel/external_magic_example.png"](https://raw.githubusercontent.com/wiki/ibm-et/spark-kernel/external_magic_example.png"){kind=link}