This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git
commit dae998147668455c2fc67218df4fca6418689e15 Author: baunsgaard <[email protected]> AuthorDate: Tue Nov 15 15:48:37 2022 +0100 [DOCS] Update Python docs --- docs/api/java/member-search-index.zip | Bin 229274 -> 229274 bytes docs/api/java/package-search-index.zip | Bin 869 -> 869 bytes docs/api/java/type-search-index.zip | Bin 14277 -> 14277 bytes docs/api/python/api/context/systemds_context.html | 15 ++++ docs/api/python/genindex.html | 2 + docs/api/python/guide/python_end_to_end_tut.html | 85 ++++++++++++++++----- docs/api/python/objects.inv | Bin 2298 -> 2302 bytes docs/api/python/searchindex.js | 2 +- .../sources/guide/python_end_to_end_tut.rst.txt | 34 ++++++--- 9 files changed, 105 insertions(+), 33 deletions(-) diff --git a/docs/api/java/member-search-index.zip b/docs/api/java/member-search-index.zip index 7d505cd54d..5ba0ae427a 100644 Binary files a/docs/api/java/member-search-index.zip and b/docs/api/java/member-search-index.zip differ diff --git a/docs/api/java/package-search-index.zip b/docs/api/java/package-search-index.zip index cfa4fe2264..04428621db 100644 Binary files a/docs/api/java/package-search-index.zip and b/docs/api/java/package-search-index.zip differ diff --git a/docs/api/java/type-search-index.zip b/docs/api/java/type-search-index.zip index 7f59f1fc64..6b2c59a496 100644 Binary files a/docs/api/java/type-search-index.zip and b/docs/api/java/type-search-index.zip differ diff --git a/docs/api/python/api/context/systemds_context.html b/docs/api/python/api/context/systemds_context.html index 0f546c3563..2050d7ba31 100644 --- a/docs/api/python/api/context/systemds_context.html +++ b/docs/api/python/api/context/systemds_context.html @@ -278,6 +278,21 @@ Afterwards capturing will be reset to the state it was before.</p> <dd><p>Close the connection to the java process and do necessary cleanup.</p> </dd></dl> +<dl class="py method"> +<dt id="systemds.context.SystemDSContext.combine"> +<code class="sig-name descname">combine</code><span class="sig-paren">(</span><em class="sig-param"><span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">Sequence<span class="p">[</span>Union<span class="p">[</span>DAGNode<span class="p">, </span>str<span class="p">, </span>int<span class="p">, </span>float<span class="p">, </span>bool<span class="p">]</span><span class="p">]</span></span></em><span class="sig-paren">)</span> → systemds.operat [...] +<dd><p>combine nodes to call compute on multiple operations.</p> +<p>This is usefull for the case of having multiple writes in one script and wanting +to execute all in one execution reusing intermediates.</p> +<p>Note this combine does not allow to return anything to the user, so if used, +please only use nodes that end with either writing or printing elements.</p> +<dl class="field-list simple"> +<dt class="field-odd">Parameters</dt> +<dd class="field-odd"><p><strong>args</strong> – A sequence that will be executed with call to compute()</p> +</dd> +</dl> +</dd></dl> + <dl class="py method"> <dt id="systemds.context.SystemDSContext.dict"> <code class="sig-name descname">dict</code><span class="sig-paren">(</span><em class="sig-param"><span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="n">Dict<span class="p">[</span>str<span class="p">, </span>Union<span class="p">[</span>DAGNode<span class="p">, </span>str<span class="p">, </span>int<span class="p">, </span>float<span class="p">, </span>bool<span class="p">]</span><span class="p">]</span></span></em><span class="sig-paren">)</span> [...] diff --git a/docs/api/python/genindex.html b/docs/api/python/genindex.html index 2bf0e31100..fd1b423e51 100644 --- a/docs/api/python/genindex.html +++ b/docs/api/python/genindex.html @@ -327,6 +327,8 @@ <li><a href="api/script_building/dag.html#systemds.script_building.dag.DAGNode.code_line">(systemds.script_building.dag.DAGNode method)</a> </li> </ul></li> + <li><a href="api/context/systemds_context.html#systemds.context.SystemDSContext.combine">combine() (systemds.context.SystemDSContext method)</a> +</li> <li><a href="api/operator/algorithms.html#systemds.operator.algorithm.components">components() (in module systemds.operator.algorithm)</a> </li> <li><a href="api/operator/node/frame.html#systemds.operator.Frame.compute">compute() (systemds.operator.Frame method)</a> diff --git a/docs/api/python/guide/python_end_to_end_tut.html b/docs/api/python/guide/python_end_to_end_tut.html index 9e17567270..634eb9b479 100644 --- a/docs/api/python/guide/python_end_to_end_tut.html +++ b/docs/api/python/guide/python_end_to_end_tut.html @@ -104,6 +104,8 @@ <li class="toctree-l3"><a class="reference internal" href="#step-2-load-the-algorithm">Step 2: Load the algorithm</a></li> <li class="toctree-l3"><a class="reference internal" href="#step-3-training-the-neural-network">Step 3: Training the neural network</a></li> <li class="toctree-l3"><a class="reference internal" href="#step-4-saving-the-model">Step 4: Saving the model</a></li> +<li class="toctree-l3"><a class="reference internal" href="#step-5-predict-on-unseen-data">Step 5: Predict on Unseen data</a></li> +<li class="toctree-l3"><a class="reference internal" href="#full-script-nn">Full Script NN</a></li> </ul> </li> </ul> @@ -342,7 +344,8 @@ for the predictions and the confusion matrix averages of each true class.</p> For this we will introduce another dml file, which can be used to train a basic feed forward network.</p> <div class="section" id="step-1-obtain-data"> <h3>Step 1: Obtain data<a class="headerlink" href="#step-1-obtain-data" title="Permalink to this headline">¶</a></h3> -<p>For the whole data setup please refer to level 1, Step 1, as these steps are identical.</p> +<p>For the whole data setup please refer to level 1, Step 1, as these steps are almost identical, +but instead of preparing the test data, we only prepare the training data.</p> <div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">systemds.context</span> <span class="kn">import</span> <span class="n">SystemDSContext</span> <span class="kn">from</span> <span class="nn">systemds.examples.tutorials.adult</span> <span class="kn">import</span> <span class="n">DataManager</span> <span class="kn">from</span> <span class="nn">systemds.operator.algorithm</span> <span class="kn">import</span> <span class="n">multiLogReg</span> @@ -365,15 +368,11 @@ For this we will introduce another dml file, which can be used to train a basic <span class="c1"># Transform frames to matrices.</span> <span class="n">X</span><span class="p">,</span> <span class="n">M1</span> <span class="o">=</span> <span class="n">X_frame</span><span class="o">.</span><span class="n">transform_encode</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_data</span><span class="p">)</span> - <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_data</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M1</span><span class="p">)</span> <span class="n">Y</span><span class="p">,</span> <span class="n">M2</span> <span class="o">=</span> <span class="n">Y_frame</span><span class="o">.</span><span class="n">transform_encode</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_labels</span><span class="p">)</span> - <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_labels</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M2</span><span class="p">)</span> <span class="c1"># Subsample to make training faster</span> <span class="n">X</span> <span class="o">=</span> <span class="n">X</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">train_count</span><span class="p">]</span> <span class="n">Y</span> <span class="o">=</span> <span class="n">Y</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">train_count</span><span class="p">]</span> - <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span> - <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span> </pre></div> </div> </div> @@ -384,12 +383,9 @@ First, we need to source the dml file for neural networks. This file includes all the necessary functions for training, evaluating, and storing the model. The returned object of the source call is further used for calling the functions. The file can be found here:</p> -<blockquote> -<div><ul class="simple"> -<li><p>:doc:tests/examples/tutorials/neural_net_source.dml</p></li> -</ul> -</div></blockquote> -<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="n">FFN_package</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">source</span><span class="p">(</span><span class="n">neural_net_src_path</span><span class="p">,</span> <span class="s2">"fnn"</span><span class="p">)</span> +<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="c1"># Load custom neural network</span> + <span class="n">neural_net_src_path</span> <span class="o">=</span> <span class="s2">"tests/examples/tutorials/neural_net_source.dml"</span> + <span class="n">FFN_package</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">source</span><span class="p">(</span><span class="n">neural_net_src_path</span><span class="p">,</span> <span class="s2">"fnn"</span><span class="p">)</span> </pre></div> </div> </div> @@ -400,7 +396,12 @@ The first two arguments are the training features and the target values we want Then we need to set the hyperparameters of the model. We choose to train for 1 epoch with a batch size of 16 and a learning rate of 0.01, which are common parameters for neural networks. The seed argument ensures that running the code again yields the same results.</p> -<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="n">network</span> <span class="o">=</span> <span class="n">FFN_package</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Y</span><span class="p">,</span> <span class="n">epochs</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">,</span> <span class="n"> [...] +<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="n">epochs</span> <span class="o">=</span> <span class="mi">1</span> + <span class="n">batch_size</span> <span class="o">=</span> <span class="mi">16</span> + <span class="n">learning_rate</span> <span class="o">=</span> <span class="mf">0.01</span> + <span class="n">seed</span> <span class="o">=</span> <span class="mi">42</span> + + <span class="n">network</span> <span class="o">=</span> <span class="n">FFN_package</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Y</span><span class="p">,</span> <span class="n">epochs</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">,</span> <span class="n">learning_rate</span><span class="p">,</span> <span class="n">seed</span><span class="p">)</span> </pre></div> </div> </div> @@ -408,12 +409,38 @@ The seed argument ensures that running the code again yields the same results.</ <h3>Step 4: Saving the model<a class="headerlink" href="#step-4-saving-the-model" title="Permalink to this headline">¶</a></h3> <p>For later usage, we can save the trained model. We only need to specify the name of our model and the file path. -This call stores the weights and biases of our model.</p> -<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span> +This call stores the weights and biases of our model. +Similarly the transformation metadata to transform input data to the model, +is saved.</p> +<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="c1"># Write metadata and trained network to disk.</span> + <span class="n">sds</span><span class="o">.</span><span class="n">combine</span><span class="p">(</span> + <span class="n">network</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/network'</span><span class="p">),</span> + <span class="n">M1</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_X'</span><span class="p">),</span> + <span class="n">M2</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_Y'</span><span class="p">)</span> + <span class="p">)</span><span class="o">.</span><span class="n">compute</span><span class="p">()</span> +</pre></div> +</div> +</div> +<div class="section" id="step-5-predict-on-unseen-data"> +<h3>Step 5: Predict on Unseen data<a class="headerlink" href="#step-5-predict-on-unseen-data" title="Permalink to this headline">¶</a></h3> +<p>Once the model is saved along with metadata, it is simple to apply it all to +unseen data:</p> +<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="c1"># Read metadata and trained network and do prediction.</span> + <span class="n">M1_r</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_X'</span><span class="p">)</span> + <span class="n">M2_r</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_Y'</span><span class="p">)</span> + <span class="n">network_r</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/network'</span><span class="p">)</span> + <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_data</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M1_r</span><span class="p">)</span> + <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_labels</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M2_r</span><span class="p">)</span> + <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span> + <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span> + <span class="n">FFN_package_2</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">source</span><span class="p">(</span><span class="n">neural_net_src_path</span><span class="p">,</span> <span class="s2">"fnn"</span><span class="p">)</span> + <span class="n">probs</span> <span class="o">=</span> <span class="n">FFN_package_2</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">Xt</span><span class="p">,</span> <span class="n">network_r</span><span class="p">)</span> + <span class="n">accuracy</span> <span class="o">=</span> <span class="n">FFN_package_2</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">probs</span><span class="p">,</span> <span class="n">Yt</span><span class="p">)</span><span class="o">.</span><span class="n">compute</span><span class="p">()</span> </pre></div> </div> -<p>Full Script NN -~~~~~~~~~~~—</p> +</div> +<div class="section" id="full-script-nn"> +<h3>Full Script NN<a class="headerlink" href="#full-script-nn" title="Permalink to this headline">¶</a></h3> <p>The complete script now can be seen here:</p> <div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">systemds.context</span> <span class="kn">import</span> <span class="n">SystemDSContext</span> <span class="kn">from</span> <span class="nn">systemds.examples.tutorials.adult</span> <span class="kn">import</span> <span class="n">DataManager</span> @@ -437,15 +464,11 @@ This call stores the weights and biases of our model.</p> <span class="c1"># Transform frames to matrices.</span> <span class="n">X</span><span class="p">,</span> <span class="n">M1</span> <span class="o">=</span> <span class="n">X_frame</span><span class="o">.</span><span class="n">transform_encode</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_data</span><span class="p">)</span> - <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_data</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M1</span><span class="p">)</span> <span class="n">Y</span><span class="p">,</span> <span class="n">M2</span> <span class="o">=</span> <span class="n">Y_frame</span><span class="o">.</span><span class="n">transform_encode</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_labels</span><span class="p">)</span> - <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_labels</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M2</span><span class="p">)</span> <span class="c1"># Subsample to make training faster</span> <span class="n">X</span> <span class="o">=</span> <span class="n">X</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">train_count</span><span class="p">]</span> <span class="n">Y</span> <span class="o">=</span> <span class="n">Y</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">train_count</span><span class="p">]</span> - <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span> - <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span> <span class="c1"># Load custom neural network</span> <span class="n">neural_net_src_path</span> <span class="o">=</span> <span class="s2">"tests/examples/tutorials/neural_net_source.dml"</span> @@ -458,7 +481,27 @@ This call stores the weights and biases of our model.</p> <span class="n">network</span> <span class="o">=</span> <span class="n">FFN_package</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Y</span><span class="p">,</span> <span class="n">epochs</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">,</span> <span class="n">learning_rate</span><span class="p">,</span> <span class="n">seed</span><span class="p">)</span> - <span class="n">network</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/'</span><span class="p">)</span><span class="o">.</span><span class="n">compute</span><span class="p">()</span> + <span class="c1"># Write metadata and trained network to disk.</span> + <span class="n">sds</span><span class="o">.</span><span class="n">combine</span><span class="p">(</span> + <span class="n">network</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/network'</span><span class="p">),</span> + <span class="n">M1</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_X'</span><span class="p">),</span> + <span class="n">M2</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_Y'</span><span class="p">)</span> + <span class="p">)</span><span class="o">.</span><span class="n">compute</span><span class="p">()</span> + + <span class="c1"># Read metadata and trained network and do prediction.</span> + <span class="n">M1_r</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_X'</span><span class="p">)</span> + <span class="n">M2_r</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/encode_Y'</span><span class="p">)</span> + <span class="n">network_r</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s1">'tests/examples/docs_test/end_to_end/network'</span><span class="p">)</span> + <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_data</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M1_r</span><span class="p">)</span> + <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt_frame</span><span class="o">.</span><span class="n">transform_apply</span><span class="p">(</span><span class="n">spec</span><span class="o">=</span><span class="n">jspec_labels</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="n">M2_r</span><span class="p">)</span> + <span class="n">Xt</span> <span class="o">=</span> <span class="n">Xt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span> + <span class="n">Yt</span> <span class="o">=</span> <span class="n">Yt</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">test_count</span><span class="p">]</span> + <span class="n">FFN_package_2</span> <span class="o">=</span> <span class="n">sds</span><span class="o">.</span><span class="n">source</span><span class="p">(</span><span class="n">neural_net_src_path</span><span class="p">,</span> <span class="s2">"fnn"</span><span class="p">)</span> + <span class="n">probs</span> <span class="o">=</span> <span class="n">FFN_package_2</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">Xt</span><span class="p">,</span> <span class="n">network_r</span><span class="p">)</span> + <span class="n">accuracy</span> <span class="o">=</span> <span class="n">FFN_package_2</span><span class="o">.</span><span class="n">eval</span><span class="p">(</span><span class="n">probs</span><span class="p">,</span> <span class="n">Yt</span><span class="p">)</span><span class="o">.</span><span class="n">compute</span><span class="p">()</span> + + <span class="kn">import</span> <span class="nn">logging</span> + <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"accuracy: "</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">accuracy</span><span class="p">))</span> </pre></div> </div> </div> diff --git a/docs/api/python/objects.inv b/docs/api/python/objects.inv index e9899fe28a..402be3e318 100644 Binary files a/docs/api/python/objects.inv and b/docs/api/python/objects.inv differ diff --git a/docs/api/python/searchindex.js b/docs/api/python/searchindex.js index 6611d6eb84..6e0d3a2fa6 100644 --- a/docs/api/python/searchindex.js +++ b/docs/api/python/searchindex.js @@ -1 +1 @@ -Search.setIndex({docnames:["api/context/systemds_context","api/operator/algorithms","api/operator/node/frame","api/operator/node/list","api/operator/node/matrix","api/operator/node/scalar","api/operator/node/source","api/operator/operation_node","api/script_building/dag","api/script_building/script","api/utils/converters","api/utils/helpers","getting_started/install","getting_started/simple_examples","guide/algorithms_basics","guide/federated","guide/python_end_to_end_tut","index"],envve [...] \ No newline at end of file +Search.setIndex({docnames:["api/context/systemds_context","api/operator/algorithms","api/operator/node/frame","api/operator/node/list","api/operator/node/matrix","api/operator/node/scalar","api/operator/node/source","api/operator/operation_node","api/script_building/dag","api/script_building/script","api/utils/converters","api/utils/helpers","getting_started/install","getting_started/simple_examples","guide/algorithms_basics","guide/federated","guide/python_end_to_end_tut","index"],envve [...] \ No newline at end of file diff --git a/docs/api/python/sources/guide/python_end_to_end_tut.rst.txt b/docs/api/python/sources/guide/python_end_to_end_tut.rst.txt index 2a8cb9fb76..961b47d61b 100644 --- a/docs/api/python/sources/guide/python_end_to_end_tut.rst.txt +++ b/docs/api/python/sources/guide/python_end_to_end_tut.rst.txt @@ -118,12 +118,13 @@ For this we will introduce another dml file, which can be used to train a basic Step 1: Obtain data ~~~~~~~~~~~~~~~~~~~ -For the whole data setup please refer to level 1, Step 1, as these steps are identical. +For the whole data setup please refer to level 1, Step 1, as these steps are almost identical, +but instead of preparing the test data, we only prepare the training data. .. include:: ../code/guide/end_to_end/part2.py :code: python :start-line: 20 - :end-line: 51 + :end-line: 47 Step 2: Load the algorithm ~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -134,12 +135,10 @@ This file includes all the necessary functions for training, evaluating, and sto The returned object of the source call is further used for calling the functions. The file can be found here: - - :doc:tests/examples/tutorials/neural_net_source.dml - .. include:: ../code/guide/end_to_end/part2.py :code: python - :start-line: 54 - :end-line: 55 + :start-line: 48 + :end-line: 51 Step 3: Training the neural network @@ -153,8 +152,8 @@ The seed argument ensures that running the code again yields the same results. .. include:: ../code/guide/end_to_end/part2.py :code: python - :start-line: 61 - :end-line: 62 + :start-line: 52 + :end-line: 58 Step 4: Saving the model @@ -163,15 +162,28 @@ Step 4: Saving the model For later usage, we can save the trained model. We only need to specify the name of our model and the file path. This call stores the weights and biases of our model. +Similarly the transformation metadata to transform input data to the model, +is saved. .. include:: ../code/guide/end_to_end/part2.py :code: python - :start-line: 64 + :start-line: 59 :end-line: 65 +Step 5: Predict on Unseen data +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once the model is saved along with metadata, it is simple to apply it all to +unseen data: + +.. include:: ../code/guide/end_to_end/part2.py + :code: python + :start-line: 66 + :end-line: 77 + Full Script NN -~~~~~~~~~~~--- +~~~~~~~~~~~~~~ The complete script now can be seen here: @@ -179,4 +191,4 @@ The complete script now can be seen here: .. include:: ../code/guide/end_to_end/part2.py :code: python :start-line: 20 - :end-line: 64 + :end-line: 80
