http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/b5b51c69/docs/v1.11/group__grp__pagerank.html ---------------------------------------------------------------------- diff --git a/docs/v1.11/group__grp__pagerank.html b/docs/v1.11/group__grp__pagerank.html new file mode 100644 index 0000000..be06283 --- /dev/null +++ b/docs/v1.11/group__grp__pagerank.html @@ -0,0 +1,329 @@ +<!-- HTML header for doxygen 1.8.4--> +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> +<html xmlns="http://www.w3.org/1999/xhtml";> +<head> +<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/> +<meta http-equiv="X-UA-Compatible" content="IE=9"/> +<meta name="generator" content="Doxygen 1.8.13"/> +<meta name="keywords" content="madlib,postgres,greenplum,machine learning,data mining,deep learning,ensemble methods,data science,market basket analysis,affinity analysis,pca,lda,regression,elastic net,huber white,proportional hazards,k-means,latent dirichlet allocation,bayes,support vector machines,svm"/> +<title>MADlib: PageRank</title> +<link href="tabs.css" rel="stylesheet" type="text/css"/> +<script type="text/javascript" src="jquery.js"></script> +<script type="text/javascript" src="dynsections.js"></script> +<link href="navtree.css" rel="stylesheet" type="text/css"/> +<script type="text/javascript" src="resize.js"></script> +<script type="text/javascript" src="navtreedata.js"></script> +<script type="text/javascript" src="navtree.js"></script> +<script type="text/javascript"> + $(document).ready(initResizable); +</script> +<link href="search/search.css" rel="stylesheet" type="text/css"/> +<script type="text/javascript" src="search/searchdata.js"></script> +<script type="text/javascript" src="search/search.js"></script> +<script type="text/javascript"> + $(document).ready(function() { init_search(); }); +</script> +<!-- hack in the navigation tree --> +<script type="text/javascript" src="eigen_navtree_hacks.js"></script> +<link href="doxygen.css" rel="stylesheet" type="text/css" /> +<link href="madlib_extra.css" rel="stylesheet" type="text/css"/> +<!-- google analytics --> +<script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + ga('create', 'UA-45382226-1', 'madlib.incubator.apache.org'); + ga('send', 'pageview'); +</script> +</head> +<body> +<div id="top"><!-- do not remove this div, it is closed by doxygen! --> +<div id="titlearea"> +<table cellspacing="0" cellpadding="0"> + <tbody> + <tr style="height: 56px;"> + <td id="projectlogo"><a href="http://madlib.incubator.apache.org";><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td> + <td style="padding-left: 0.5em;"> + <div id="projectname"> + <span id="projectnumber">1.11</span> + </div> + <div id="projectbrief">User Documentation for MADlib</div> + </td> + <td> <div id="MSearchBox" class="MSearchBoxInactive"> + <span class="left"> + <img id="MSearchSelect" src="search/mag_sel.png" + onmouseover="return searchBox.OnSearchSelectShow()" + onmouseout="return searchBox.OnSearchSelectHide()" + alt=""/> + <input type="text" id="MSearchField" value="Search" accesskey="S" + onfocus="searchBox.OnSearchFieldFocus(true)" + onblur="searchBox.OnSearchFieldFocus(false)" + onkeyup="searchBox.OnSearchFieldChange(event)"/> + </span><span class="right"> + <a id="MSearchClose" href="javascript:searchBox.CloseResultsWindow()"><img id="MSearchCloseImg" border="0" src="search/close.png" alt=""/></a> + </span> + </div> +</td> + </tr> + </tbody> +</table> +</div> +<!-- end header part --> +<!-- Generated by Doxygen 1.8.13 --> +<script type="text/javascript"> +var searchBox = new SearchBox("searchBox", "search",false,'Search'); +</script> +</div><!-- top --> +<div id="side-nav" class="ui-resizable side-nav-resizable"> + <div id="nav-tree"> + <div id="nav-tree-contents"> + <div id="nav-sync" class="sync"></div> + </div> + </div> + <div id="splitbar" style="-moz-user-select:none;" + class="ui-resizable-handle"> + </div> +</div> +<script type="text/javascript"> +$(document).ready(function(){initNavTree('group__grp__pagerank.html','');}); +</script> +<div id="doc-content"> +<!-- window showing the filter options --> +<div id="MSearchSelectWindow" + onmouseover="return searchBox.OnSearchSelectShow()" + onmouseout="return searchBox.OnSearchSelectHide()" + onkeydown="return searchBox.OnSearchSelectKey(event)"> +</div> + +<!-- iframe showing the search results (closed by default) --> +<div id="MSearchResultsWindow"> +<iframe src="javascript:void(0)" frameborder="0" + name="MSearchResults" id="MSearchResults"> +</iframe> +</div> + +<div class="header"> + <div class="headertitle"> +<div class="title">PageRank<div class="ingroups"><a class="el" href="group__grp__graph.html">Graph</a></div></div> </div> +</div><!--header--> +<div class="contents"> +<div class="toc"><b>Contents</b> <ul> +<li> +<a href="#pagerank">PageRank</a> </li> +<li> +<a href="#notes">Notes</a> </li> +<li> +<a href="#examples">Examples</a> </li> +<li> +<a href="#literature">Literature</a> </li> +</ul> +</div><p>Given a graph, the PageRank algorithm outputs a probability distribution representing the likelihood that a person randomly traversing the graph will arrive at any particular vertex. This algorithm was originally used by Google to rank websites where the World Wide Web was modeled as a directed graph with the vertices representing the websites.</p> +<p><a class="anchor" id="pagerank"></a></p><dl class="section user"><dt>PageRank</dt><dd><pre class="syntax"> +pagerank( vertex_table, + vertex_id, + edge_table, + edge_args, + out_table, + damping_factor, + max_iter, + threshold, + grouping_cols + ) +</pre></dd></dl> +<p><b>Arguments</b> </p><dl class="arglist"> +<dt>vertex_table </dt> +<dd><p class="startdd">TEXT. Name of the table containing the vertex data for the graph. Must contain the column specified in the 'vertex_id' parameter below.</p> +<p class="enddd"></p> +</dd> +<dt>vertex_id </dt> +<dd><p class="startdd">TEXT, default = 'id'. Name of the column in 'vertex_table' containing vertex ids. The vertex ids are of type INTEGER with no duplicates. They do not need to be contiguous.</p> +<p class="enddd"></p> +</dd> +<dt>edge_table </dt> +<dd><p class="startdd">TEXT. Name of the table containing the edge data. The edge table must contain columns for source vertex and destination vertex.</p> +<p class="enddd"></p> +</dd> +<dt>edge_args </dt> +<dd><p class="startdd">TEXT. A comma-delimited string containing multiple named arguments of the form "name=value". The following parameters are supported for this string argument:</p><ul> +<li>src (INTEGER): Name of the column containing the source vertex ids in the edge table. Default column name is 'src'.</li> +<li>dest (INTEGER): Name of the column containing the destination vertex ids in the edge table. Default column name is 'dest'.</li> +</ul> +<p class="enddd"></p> +</dd> +<dt>out_table </dt> +<dd><p class="startdd">TEXT. Name of the table to store the result of PageRank. It will contain a row for every vertex from 'vertex_table' with the following columns:</p><ul> +<li>vertex_id : The id of a vertex. Will use the input parameter 'vertex_id' for column naming.</li> +<li>pagerank : The vertex's PageRank.</li> +<li>grouping_cols : Grouping column (if any) values associated with the vertex_id.</li> +</ul> +<p>A summary table is also created that contains information regarding the number of iterations required for convergence. It is named by adding the suffix '_summary' to the 'out_table' parameter.</p> +<p class="enddd"></p> +</dd> +<dt>damping_factor </dt> +<dd><p class="startdd">FLOAT8, default 0.85. The probability, at any step, that a user will continue following the links in a random surfer model.</p> +<p class="enddd"></p> +</dd> +<dt>max_iter </dt> +<dd><p class="startdd">INTEGER, default: 100. The maximum number of iterations allowed.</p> +<p class="enddd"></p> +</dd> +<dt>threshold </dt> +<dd><p class="startdd">FLOAT8, default: (1/number of vertices * 100). If the difference between the PageRank of every vertex of two consecutive iterations is smaller than 'threshold', or the iteration number is larger than 'max_iter', the computation stops. If you set the threshold to zero, then you will force the algorithm to run for the full number of iterations specified in 'max_iter'. It is advisable to set threshold to a value lower than 1/(number of vertices in the graph) since the PageRank value of nodes is initialized to that value.</p> +<p class="enddd"></p> +</dd> +<dt>grouping_cols (optional) </dt> +<dd>TEXT, default: NULL. A single column or a list of comma-separated columns that divides the input data into discrete groups, resulting in one distribution per group. When this value is NULL, no grouping is used and a single model is generated for all data. <dl class="section note"><dt>Note</dt><dd>Expressions are not currently supported for 'grouping_cols'.</dd></dl> +</dd> +</dl> +<p><a class="anchor" id="notes"></a></p><dl class="section user"><dt>Notes</dt><dd></dd></dl> +<p>The PageRank algorithm proposed by Larry Page and Sergey Brin is used [1].</p> +<p><a class="anchor" id="examples"></a></p><dl class="section user"><dt>Examples</dt><dd></dd></dl> +<ol type="1"> +<li>Create vertex and edge tables to represent the graph: <pre class="syntax"> +DROP TABLE IF EXISTS vertex, edge; +CREATE TABLE vertex( + id INTEGER + ); +CREATE TABLE edge( + src INTEGER, + dest INTEGER, + user_id INTEGER + ); +INSERT INTO vertex VALUES +(0), +(1), +(2), +(3), +(4), +(5), +(6); +INSERT INTO edge VALUES +(0, 1, 1), +(0, 2, 1), +(0, 4, 1), +(1, 2, 1), +(1, 3, 1), +(2, 3, 1), +(2, 5, 1), +(2, 6, 1), +(3, 0, 1), +(4, 0, 1), +(5, 6, 1), +(6, 3, 1), +(0, 1, 2), +(0, 2, 2), +(0, 4, 2), +(1, 2, 2), +(1, 3, 2), +(2, 3, 2), +(3, 0, 2), +(4, 0, 2), +(5, 6, 2), +(6, 3, 2); +</pre></li> +<li>Compute the PageRank: <pre class="syntax"> +DROP TABLE IF EXISTS pagerank_out, pagerank_out_summary; +SELECT madlib.pagerank( + 'vertex', -- Vertex table + 'id', -- Vertix id column + 'edge', -- Edge table + 'src=src, dest=dest', -- Comma delimted string of edge arguments + 'pagerank_out'); -- Output table of PageRank +SELECT * FROM pagerank_out ORDER BY pagerank DESC; +</pre> <pre class="result"> + id | pagerank +----+------------------- + 0 | 0.28753749341184 + 3 | 0.21016988901855 + 2 | 0.14662683454062 + 4 | 0.10289614384217 + 1 | 0.10289614384217 + 6 | 0.09728637768887 + 5 | 0.05258711765692 +(7 rows) +</pre> <pre class="syntax"> +SELECT * FROM pagerank_out_summary; +</pre> <pre class="result"> + __iterations__ + ----------------+ + 16 +(1 row) +</pre></li> +<li>Running PageRank with a damping factor of 0.5 results in different final values: <pre class="syntax"> +DROP TABLE IF EXISTS pagerank_out, pagerank_out_summary; +SELECT madlib.pagerank( + 'vertex', -- Vertex table + 'id', -- Vertix id column + 'edge', -- Edge table + 'src=src, dest=dest', -- Comma delimted string of edge arguments + 'pagerank_out', -- Output table of PageRank + 0.5); -- Damping factor +SELECT * FROM pagerank_out ORDER BY pagerank DESC; +</pre> <pre class="result"> + id | pagerank +----+-------------------- + 0 | 0.225477161441199 + 3 | 0.199090328586664 + 2 | 0.136261327206477 + 6 | 0.132691559968224 + 4 | 0.109009291409508 + 1 | 0.109009291409508 + 5 | 0.0884610399788161 +(7 rows) +</pre></li> +<li>Now compute the PageRank of vertices associated with each user using the grouping feature: <pre class="syntax"> +DROP TABLE IF EXISTS pagerank_out, pagerank_out_summary; +SELECT madlib.pagerank( + 'vertex', -- Vertex table + 'id', -- Vertix id column + 'edge', -- Edge table + 'src=src, dest=dest', -- Comma delimted string of edge arguments + 'pagerank_out', -- Output table of PageRank + NULL, -- Default damping factor (0.85) + NULL, -- Default max iters (100) + 0.00000001, -- Threshold + 'user_id'); -- Grouping column name +SELECT * FROM pagerank_out ORDER BY user_id, pagerank DESC; +</pre> <pre class="result"> + user_id | id | pagerank +---------+----+-------------------- + 1 | 0 | 0.27825488388552 + 1 | 3 | 0.20188114667075 + 1 | 2 | 0.14288112346059 + 1 | 6 | 0.11453637832147 + 1 | 1 | 0.10026745615438 + 1 | 4 | 0.10026745615438 + 1 | 5 | 0.06191155535288 + 2 | 0 | 0.31854625004173 + 2 | 3 | 0.23786686773343 + 2 | 2 | 0.15914876489397 + 2 | 1 | 0.11168334437971 + 2 | 4 | 0.11168334437971 + 2 | 6 | 0.03964285714285 + 2 | 5 | 0.02142857142857 +(14 rows) +</pre> <pre class="syntax"> +SELECT * FROM pagerank_out_summary ORDER BY user_id; +</pre> <pre class="result"> + user_id | __iterations__ +---------+---------------- + 1 | 27 + 2 | 31 +(2 rows) +</pre></li> +</ol> +<p><a class="anchor" id="literature"></a></p><dl class="section user"><dt>Literature</dt><dd></dd></dl> +<p>[1] PageRank algorithm. <a href="https://en.wikipedia.org/wiki/PageRank";>https://en.wikipedia.org/wiki/PageRank</a> </p> +</div><!-- contents --> +</div><!-- doc-content --> +<!-- start footer part --> +<div id="nav-path" class="navpath"><!-- id is needed for treeview function! --> + <ul> + <li class="footer">Generated on Tue May 16 2017 13:24:38 for MADlib by + <a href="http://www.doxygen.org/index.html";> + <img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.13 </li> + </ul> +</div> +</body> +</html>
http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/b5b51c69/docs/v1.11/group__grp__path.html ---------------------------------------------------------------------- diff --git a/docs/v1.11/group__grp__path.html b/docs/v1.11/group__grp__path.html new file mode 100644 index 0000000..576e6cb --- /dev/null +++ b/docs/v1.11/group__grp__path.html @@ -0,0 +1,473 @@ +<!-- HTML header for doxygen 1.8.4--> +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> +<html xmlns="http://www.w3.org/1999/xhtml";> +<head> +<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/> +<meta http-equiv="X-UA-Compatible" content="IE=9"/> +<meta name="generator" content="Doxygen 1.8.13"/> +<meta name="keywords" content="madlib,postgres,greenplum,machine learning,data mining,deep learning,ensemble methods,data science,market basket analysis,affinity analysis,pca,lda,regression,elastic net,huber white,proportional hazards,k-means,latent dirichlet allocation,bayes,support vector machines,svm"/> +<title>MADlib: Path</title> +<link href="tabs.css" rel="stylesheet" type="text/css"/> +<script type="text/javascript" src="jquery.js"></script> +<script type="text/javascript" src="dynsections.js"></script> +<link href="navtree.css" rel="stylesheet" type="text/css"/> +<script type="text/javascript" src="resize.js"></script> +<script type="text/javascript" src="navtreedata.js"></script> +<script type="text/javascript" src="navtree.js"></script> +<script type="text/javascript"> + $(document).ready(initResizable); +</script> +<link href="search/search.css" rel="stylesheet" type="text/css"/> +<script type="text/javascript" src="search/searchdata.js"></script> +<script type="text/javascript" src="search/search.js"></script> +<script type="text/javascript"> + $(document).ready(function() { init_search(); }); +</script> +<!-- hack in the navigation tree --> +<script type="text/javascript" src="eigen_navtree_hacks.js"></script> +<link href="doxygen.css" rel="stylesheet" type="text/css" /> +<link href="madlib_extra.css" rel="stylesheet" type="text/css"/> +<!-- google analytics --> +<script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + ga('create', 'UA-45382226-1', 'madlib.incubator.apache.org'); + ga('send', 'pageview'); +</script> +</head> +<body> +<div id="top"><!-- do not remove this div, it is closed by doxygen! --> +<div id="titlearea"> +<table cellspacing="0" cellpadding="0"> + <tbody> + <tr style="height: 56px;"> + <td id="projectlogo"><a href="http://madlib.incubator.apache.org";><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td> + <td style="padding-left: 0.5em;"> + <div id="projectname"> + <span id="projectnumber">1.11</span> + </div> + <div id="projectbrief">User Documentation for MADlib</div> + </td> + <td> <div id="MSearchBox" class="MSearchBoxInactive"> + <span class="left"> + <img id="MSearchSelect" src="search/mag_sel.png" + onmouseover="return searchBox.OnSearchSelectShow()" + onmouseout="return searchBox.OnSearchSelectHide()" + alt=""/> + <input type="text" id="MSearchField" value="Search" accesskey="S" + onfocus="searchBox.OnSearchFieldFocus(true)" + onblur="searchBox.OnSearchFieldFocus(false)" + onkeyup="searchBox.OnSearchFieldChange(event)"/> + </span><span class="right"> + <a id="MSearchClose" href="javascript:searchBox.CloseResultsWindow()"><img id="MSearchCloseImg" border="0" src="search/close.png" alt=""/></a> + </span> + </div> +</td> + </tr> + </tbody> +</table> +</div> +<!-- end header part --> +<!-- Generated by Doxygen 1.8.13 --> +<script type="text/javascript"> +var searchBox = new SearchBox("searchBox", "search",false,'Search'); +</script> +</div><!-- top --> +<div id="side-nav" class="ui-resizable side-nav-resizable"> + <div id="nav-tree"> + <div id="nav-tree-contents"> + <div id="nav-sync" class="sync"></div> + </div> + </div> + <div id="splitbar" style="-moz-user-select:none;" + class="ui-resizable-handle"> + </div> +</div> +<script type="text/javascript"> +$(document).ready(function(){initNavTree('group__grp__path.html','');}); +</script> +<div id="doc-content"> +<!-- window showing the filter options --> +<div id="MSearchSelectWindow" + onmouseover="return searchBox.OnSearchSelectShow()" + onmouseout="return searchBox.OnSearchSelectHide()" + onkeydown="return searchBox.OnSearchSelectKey(event)"> +</div> + +<!-- iframe showing the search results (closed by default) --> +<div id="MSearchResultsWindow"> +<iframe src="javascript:void(0)" frameborder="0" + name="MSearchResults" id="MSearchResults"> +</iframe> +</div> + +<div class="header"> + <div class="headertitle"> +<div class="title">Path<div class="ingroups"><a class="el" href="group__grp__utility__functions.html">Utility Functions</a></div></div> </div> +</div><!--header--> +<div class="contents"> +<div class="toc"><b>Contents</b> <ul> +<li> +<a href="#syntax">Function Syntax</a> </li> +<li> +<a href="#examples">Examples</a> </li> +<li> +<a href="#nomenclature">Nomenclature</a> </li> +<li> +<a href="#literature">Literature</a> </li> +</ul> +</div><p>The goal of the MADlib path function is to perform regular pattern matching over a sequence of rows, and to extract useful information about the pattern matches. The useful information could be a simple count of matches or something more involved like aggregations or window functions.</p> +<p>Symbols are used to identify particular rows of interest. Then, standard PostgreSQL pattern matching using symbols can be applied to identify patterns across the rows of interest. (This is similar in concept to regular expressions which match patterns within strings of text.)</p> +<p>For example, a symbol can be defined for purchase events by on-line shoppers. Then, preceding events that led to the purchase can be identified and operated on, perhaps to find the common actions that resulted in a purchase. Or conversely, to find actions that resulted in an exit without a purchase having been made.</p> +<p>Steps on how to use path functions:</p> +<ol type="1"> +<li>Partition input rows.</li> +<li>Order the partitions.</li> +<li>Define symbols to match rows of interest.</li> +<li>Define regular expression of symbols and operators to define patterns to match in your ordered partitions.</li> +<li>Define an aggregate function to compute for each pattern match.</li> +<li>If desired, output the pattern matches for inspection or to operate on them with subsequent queries.</li> +</ol> +<p><a class="anchor" id="syntax"></a></p><dl class="section user"><dt>Function Syntax</dt><dd><pre class="syntax"> +path( + source_table, + output_table, + partition_expr, + order_expr, + symbol, + pattern, + aggregate_func, + persist_rows, + overlapping_patterns +) +</pre></dd></dl> +<p><b>Arguments</b> </p><dl class="arglist"> +<dt>source_table </dt> +<dd><p class="startdd">VARCHAR. Name of the source table, containing data for path analysis.</p> +<p class="enddd"></p> +</dd> +<dt>output_table </dt> +<dd><p class="startdd">VARCHAR. Name of the result table.</p> +<p class="enddd"></p> +</dd> +<dt>partition_expr </dt> +<dd><p class="startdd">VARCHAR. The 'partition_expr' can be a single column or a list of comma-separated columns/expressions to divide all rows into groups, or partitions. Matching is applied across the rows that fall into the same partition. This can be NULL or '' to indicate the matching is to be applied to the whole table.</p> +<p class="enddd"></p> +</dd> +<dt>order_expr </dt> +<dd><p class="startdd">VARCHAR. This expression controls the order in which rows are processed or matched in a partition. For example, time is a common way to order partitions. </p> +<p class="enddd"></p> +</dd> +<dt>symbol </dt> +<dd><p class="startdd">VARCHAR. Symbols enable you to express patterns of interest in a simple way (see definition of âpatternâ argument below). A symbol identifies a row of a particular type that youâre searching for as part of a pattern match. Symbol definition uses the standard PostgreSQL assignment statement 'identifier := expression;' [1]. A given row can only match one symbol. If a row matches multiple symbols, the symbol that comes first in the symbol definition list will take precedence. </p> +<p class="enddd"></p> +</dd> +<dt>pattern </dt> +<dd><p class="startdd">VARCHAR. The 'pattern' clause defines the pattern that the path algorithm searches for. You express the pattern using symbols and operators following regular PostgreSQL pattern matching syntax and rules [2].</p> +<p><a class="anchor" id="note"></a></p><dl class="section note"><dt>Note</dt><dd>Symbols defined using more than one (1) character need to be enclosed in parentheses '()' when referenced in the 'pattern' argument. For example:<ul> +<li>a symbol defined as 'a' in the 'symbol' argument can be used directly in the 'pattern' argument</li> +<li>a symbol defined as 'abc' in the 'symbol' argument must be written as '(abc)' in the 'pattern' argument</li> +</ul> +</dd></dl> +<p>The following pattern matching metacharacters are supported: </p><ul> +<li> +| denotes alternation (either of two alternatives). </li> +<li> +? denotes repetition of the previous item zero or one time. </li> +<li> +* denotes repetition of the previous item zero or more times. </li> +<li> ++ denotes repetition of the previous item one or more times. </li> +<li> +{m} denotes repetition of the previous item exactly m times. </li> +<li> +{m,} denotes repetition of the previous item m or more times. </li> +<li> +{m,n} denotes repetition of the previous item at least m and not more than n times. </li> +<li> +Parentheses () can be used to group items into a single logical item. </li> +</ul> +<p class="enddd"></p> +</dd> +<dt>aggregate_func (optional) </dt> +<dd><p class="startdd">VARCHAR, default NULL. A comma-separated list of aggregates to be applied to the pattern matches [3]. Please note that window functions cannot currently be used in the parameter 'aggregate_func'. If you want to use a window function [4], output the pattern matches and write a SQL query with a window function over the output tuples (see 'persist_rows' parameter below).</p> +<p>If you just want to output the pattern matched rows and not compute any aggregates, you can put NULL or '' in the 'aggregate_func' parameter. </p> +<p class="enddd"></p> +</dd> +<dt>persist_rows (optional) </dt> +<dd><p class="startdd">BOOLEAN, default FALSE. If TRUE the matched rows are persisted in a separate output table. This table is named as <output_table>_tuples (the string "_tuples" is added as suffix to the value of <em>output_table</em>). </p> +<p class="enddd"></p> +</dd> +<dt>overlapping_patterns (optional) </dt> +<dd><p class="startdd">BOOLEAN, default FALSE. If TRUE find every occurrence of the pattern in the partition, regardless of whether it might have been part of a previously found match. </p> +<p class="enddd"></p> +</dd> +</dl> +<p><a class="anchor" id="examples"></a></p><dl class="section user"><dt>Examples</dt><dd></dd></dl> +<p>The data set describes shopper behavior on a notional web site that sells beer and wine. A beacon fires an event to a log file when the shopper visits different pages on the site: landing page, beer selection page, wine selection page, and checkout. Other pages on the site like help pages show up in the logs as well. Letâs assume that the log has been sessionized.</p> +<ol type="1"> +<li>Create the date table: <pre class="example"> +DROP TABLE IF EXISTS eventlog; +CREATE TABLE eventlog (event_timestamp TIMESTAMP, + user_id INT, + session_id INT, + page TEXT, + revenue FLOAT); +INSERT INTO eventlog VALUES +('04/15/2015 01:03:00', 100821, 100, 'LANDING', 0), +('04/15/2015 01:04:00', 100821, 100, 'WINE', 0), +('04/15/2015 01:05:00', 100821, 100, 'CHECKOUT', 39), +('04/15/2015 02:06:00', 100821, 101, 'WINE', 0), +('04/15/2015 02:09:00', 100821, 101, 'WINE', 0), +('04/15/2015 01:15:00', 101121, 102, 'LANDING', 0), +('04/15/2015 01:16:00', 101121, 102, 'WINE', 0), +('04/15/2015 01:17:00', 101121, 102, 'CHECKOUT', 15), +('04/15/2015 01:18:00', 101121, 102, 'LANDING', 0), +('04/15/2015 01:19:00', 101121, 102, 'HELP', 0), +('04/15/2015 01:21:00', 101121, 102, 'WINE', 0), +('04/15/2015 01:22:00', 101121, 102, 'CHECKOUT', 23), +('04/15/2015 02:15:00', 101331, 103, 'LANDING', 0), +('04/15/2015 02:16:00', 101331, 103, 'WINE', 0), +('04/15/2015 02:17:00', 101331, 103, 'HELP', 0), +('04/15/2015 02:18:00', 101331, 103, 'WINE', 0), +('04/15/2015 02:19:00', 101331, 103, 'CHECKOUT', 16), +('04/15/2015 02:22:00', 101443, 104, 'BEER', 0), +('04/15/2015 02:25:00', 101443, 104, 'CHECKOUT', 12), +('04/15/2015 02:29:00', 101881, 105, 'LANDING', 0), +('04/15/2015 02:30:00', 101881, 105, 'BEER', 0), +('04/15/2015 01:05:00', 102201, 106, 'LANDING', 0), +('04/15/2015 01:06:00', 102201, 106, 'HELP', 0), +('04/15/2015 01:09:00', 102201, 106, 'LANDING', 0), +('04/15/2015 02:15:00', 102201, 107, 'WINE', 0), +('04/15/2015 02:16:00', 102201, 107, 'BEER', 0), +('04/15/2015 02:17:00', 102201, 107, 'WINE', 0), +('04/15/2015 02:18:00', 102871, 108, 'BEER', 0), +('04/15/2015 02:19:00', 102871, 108, 'WINE', 0), +('04/15/2015 02:22:00', 102871, 108, 'CHECKOUT', 21), +('04/15/2015 02:25:00', 102871, 108, 'LANDING', 0), +('04/15/2015 02:17:00', 103711, 109, 'BEER', 0), +('04/15/2015 02:18:00', 103711, 109, 'LANDING', 0), +('04/15/2015 02:19:00', 103711, 109, 'WINE', 0); +</pre></li> +<li>Calculate the revenue by checkout: <pre class="example"> +DROP TABLE IF EXISTS path_output, path_output_tuples; +SELECT madlib.path( + 'eventlog', -- Name of input table + 'path_output', -- Table name to store path results + 'session_id', -- Partition input table by session + 'event_timestamp ASC', -- Order partitions in input table by time + 'buy:=page=''CHECKOUT''', -- Define a symbol for checkout events + '(buy)', -- Pattern search: purchase + 'sum(revenue) as checkout_rev', -- Aggregate: sum revenue by checkout + TRUE -- Persist matches + ); +SELECT * FROM path_output ORDER BY session_id, match_id; +</pre> Result: <pre class="result"> + session_id | match_id | checkout_rev +------------+----------+-------------- + 100 | 1 | 39 + 102 | 1 | 15 + 102 | 2 | 23 + 103 | 1 | 16 + 104 | 1 | 12 + 108 | 1 | 21 +(6 rows) +</pre> Note that there are 2 checkouts within session 102, which is apparent from the 'match_id' column. This serves to illustrate that the 'aggregate_func' operates on a <em>per pattern match</em> basis, not on a <em>per partition</em> basis. If in fact we wanted revenue by partition ('session_id' in this example), then we could do: <pre class="example"> +SELECT session_id, sum(checkout_rev) FROM path_output GROUP BY session_id ORDER BY session_id; +</pre> Result: <pre class="result"> + session_id | sum +------------+----- + 100 | 39 + 102 | 38 + 103 | 16 + 104 | 12 + 108 | 21 +(5 rows) +</pre> Since we set TRUE for 'persist_rows', we can view the associated pattern matches: <pre class="example"> +SELECT * FROM path_output_tuples ORDER BY session_id ASC, event_timestamp ASC; +</pre> Result: <pre class="result"> + event_timestamp | user_id | session_id | page | revenue | symbol | match_id +---------------------+---------+------------+----------+---------+--------+---------- + 2015-04-15 01:05:00 | 100821 | 100 | CHECKOUT | 39 | buy | 1 + 2015-04-15 01:17:00 | 101121 | 102 | CHECKOUT | 15 | buy | 1 + 2015-04-15 01:22:00 | 101121 | 102 | CHECKOUT | 23 | buy | 2 + 2015-04-15 02:19:00 | 101331 | 103 | CHECKOUT | 16 | buy | 1 + 2015-04-15 02:25:00 | 101443 | 104 | CHECKOUT | 12 | buy | 1 + 2015-04-15 02:22:00 | 102871 | 108 | CHECKOUT | 21 | buy | 1 +(6 rows) +</pre> Notice that the 'symbol' and 'match_id' columns are added to the right of the matched rows.</li> +<li>We are interested in sessions with an order placed within 4 pages of entering the shopping site via the landing page. We represent this by the regular expression: '(land)[^(land)(buy)]{0,2}(buy)'. In other words, visit to the landing page followed by from 0 to 2 non-entry, non-sale pages, followed by a purchase. The SQL is as follows: <pre class="example"> +DROP TABLE IF EXISTS path_output, path_output_tuples; +SELECT madlib.path( + 'eventlog', -- Name of input table + 'path_output', -- Table name to store path results + 'session_id', -- Partition input table by session + 'event_timestamp ASC', -- Order partitions in input table by time + 'land:=page=''LANDING'', + wine:=page=''WINE'', + beer:=page=''BEER'', + buy:=page=''CHECKOUT'', + other:=page<>''LANDING'' AND page<>''WINE'' AND page<>''BEER'' AND page<>''CHECKOUT''', -- Symbols for page types + '(land)[^(land)(buy)]{0,2}(buy)', -- Purchase within 4 pages entering site + 'sum(revenue) as checkout_rev', -- Aggregate: sum revenue by checkout + TRUE -- Persist matches + ); +SELECT * FROM path_output ORDER BY session_id, match_id; +</pre> Result: <pre class="result"> + session_id | match_id | session_rev +------------+----------+------------- + 100 | 1 | 39 + 102 | 1 | 15 + 102 | 2 | 23 +(3 rows) +</pre> Now view the associated pattern matches: <pre class="example"> +SELECT * FROM path_output_tuples ORDER BY session_id ASC, event_timestamp ASC; +</pre> Result: <pre class="result"> + event_timestamp | user_id | session_id | page | revenue | symbol | match_id +---------------------+---------+------------+----------+---------+--------+---------- + 2015-04-15 01:03:00 | 100821 | 100 | LANDING | 0 | land | 1 + 2015-04-15 01:04:00 | 100821 | 100 | WINE | 0 | wine | 1 + 2015-04-15 01:05:00 | 100821 | 100 | CHECKOUT | 39 | buy | 1 + 2015-04-15 01:15:00 | 101121 | 102 | LANDING | 0 | land | 1 + 2015-04-15 01:16:00 | 101121 | 102 | WINE | 0 | wine | 1 + 2015-04-15 01:17:00 | 101121 | 102 | CHECKOUT | 15 | buy | 1 + 2015-04-15 01:18:00 | 101121 | 102 | LANDING | 0 | land | 2 + 2015-04-15 01:19:00 | 101121 | 102 | HELP | 0 | other | 2 + 2015-04-15 01:21:00 | 101121 | 102 | WINE | 0 | wine | 2 + 2015-04-15 01:22:00 | 101121 | 102 | CHECKOUT | 23 | buy | 2 +(10 rows) +</pre></li> +<li>We may want to use a window function instead of an aggregate. Currently, only aggregates are supported in the core path function in the parameter 'aggregate_func'. However, you can write window functions on the output tuples to achieve the desired result.   Continuing the previous example, letâs say we want to compute average revenue for checkouts within 4 pages of entering the shopping site via the landing page: <pre class="example"> +SELECT DATE(event_timestamp), user_id, session_id, revenue, + avg(revenue) OVER (PARTITION BY DATE(event_timestamp)) as avg_checkout_rev + FROM path_output_tuples + WHERE page='CHECKOUT' + ORDER BY user_id, session_id; +</pre> Result: <pre class="result"> + date | user_id | session_id | revenue | avg_checkout_rev +------------+---------+------------+---------+------------------ + 2015-04-15 | 100821 | 100 | 39 | 25.6666666666667 + 2015-04-15 | 101121 | 102 | 15 | 25.6666666666667 + 2015-04-15 | 101121 | 102 | 23 | 25.6666666666667 +(3 rows) +</pre> Here we are partitioning the window function by day because we want daily averages, although our sample data set only has a single day.</li> +<li>Now we want to do a golden path analysis to find the most successful shopper paths through the site. Since our data set is small, we decide this means the most frequently viewed page just before a checkout is made: <pre class="example"> +DROP TABLE IF EXISTS path_output, path_output_tuples; +SELECT madlib.path( + 'eventlog', -- Name of input table + 'path_output', -- Table name to store path results + 'session_id', -- Partition input table by session + 'event_timestamp ASC', -- Order partitions in input table by time + 'land:=page=''LANDING'', + wine:=page=''WINE'', + beer:=page=''BEER'', + buy:=page=''CHECKOUT'', + other:=page<>''LANDING'' AND page<>''WINE'' AND page<>''BEER'' AND page<>''CHECKOUT''', -- Symbols for page types + '[^(buy)](buy)', -- Pattern to match + 'array_agg(page ORDER BY session_id ASC, event_timestamp ASC) as page_path', -- Build array with shopper paths + FALSE -- Don't persist matches + ); +</pre> Now count the common paths and print the most frequent: <pre class="example"> +SELECT count(*), page_path from + (SELECT * FROM path_output) q +GROUP BY page_path +ORDER BY count(*) DESC +LIMIT 10; +</pre> Result: <pre class="result"> + count | page_path +-------+----------------- + 5 | {WINE,CHECKOUT} + 1 | {BEER,CHECKOUT} +(2 rows) +</pre> There are only 2 different paths. The wine page is viewed more frequently than the beer page just before checkout.</li> +<li>To demonstrate the use of 'overlapping_patterns', consider a pattern with at least one page followed by and ending with a checkout: <pre class="example"> +DROP TABLE IF EXISTS path_output, path_output_tuples; +SELECT madlib.path( + 'eventlog', -- Name of the table + 'path_output', -- Table name to store the path results + 'session_id', -- Partition by session + 'event_timestamp ASC', -- Order partitions in input table by time + $$ nobuy:=page<>'CHECKOUT', + buy:=page='CHECKOUT' + $$, -- Definition of symbols used in the pattern definition + '(nobuy)+(buy)', -- At least one page followed by and ending with a CHECKOUT. + 'array_agg(page ORDER BY session_id ASC, event_timestamp ASC) as page_path', + FALSE, -- Don't persist matches + TRUE -- Turn on overlapping patterns + ); +SELECT * FROM path_output ORDER BY session_id, match_id; +</pre> Result with overlap turned on: <pre class="result"> + session_id | match_id | page_path +------------+----------+----------------------------------- + 100 | 1 | {LANDING,WINE,CHECKOUT} + 100 | 2 | {WINE,CHECKOUT} + 102 | 1 | {LANDING,WINE,CHECKOUT} + 102 | 2 | {WINE,CHECKOUT} + 102 | 3 | {LANDING,HELP,WINE,CHECKOUT} + 102 | 4 | {HELP,WINE,CHECKOUT} + 102 | 5 | {WINE,CHECKOUT} + 103 | 1 | {LANDING,WINE,HELP,WINE,CHECKOUT} + 103 | 2 | {WINE,HELP,WINE,CHECKOUT} + 103 | 3 | {HELP,WINE,CHECKOUT} + 103 | 4 | {WINE,CHECKOUT} + 104 | 1 | {BEER,CHECKOUT} + 108 | 1 | {BEER,WINE,CHECKOUT} + 108 | 2 | {WINE,CHECKOUT} +(14 rows) +</pre> With overlap turned off, the result would be: <pre class="result"> + session_id | match_id | page_path +------------+----------+----------------------------------- + 100 | 1 | {LANDING,WINE,CHECKOUT} + 102 | 1 | {LANDING,WINE,CHECKOUT} + 102 | 2 | {LANDING,HELP,WINE,CHECKOUT} + 103 | 1 | {LANDING,WINE,HELP,WINE,CHECKOUT} + 104 | 1 | {BEER,CHECKOUT} + 108 | 1 | {BEER,WINE,CHECKOUT} +(6 rows) +</pre></li> +</ol> +<p><a class="anchor" id="note"></a></p><dl class="section note"><dt>Note</dt><dd>Please note some current limitations of the path algorithm.<ul> +<li>Window functions cannot currently be used in the parameter 'aggregate_func'. Instead, output the pattern matches and write a SQL query with a window function over the output tuples.</li> +<li>A given row can only match one symbol. If a row matches multiple symbols, the symbol that comes <em>first</em> in the symbol definition list will take precedence.</li> +<li>Maximum number of symbols that can be defined is 35.</li> +<li>The columns 'match_id' and 'symbol' are generated by the path algorithm. If coincidently you have columns in your input data named 'match_id' or 'symbol', the system generated column names will be changed to "__madlib_path_match_id__" and "__madlib_path_symbol__"</li> +</ul> +</dd></dl> +<p><a class="anchor" id="nomenclature"></a></p><dl class="section user"><dt>Nomenclature</dt><dd></dd></dl> +<p>Partition</p><ul> +<li>scope of rows to be searched for pattern match</li> +<li>typical examples: user id, session id, portfolio id</li> +</ul> +<p>Order</p><ul> +<li>sort order of input rows in partition</li> +<li>typical example: time</li> +</ul> +<p>Symbol</p><ul> +<li>a row of a particular type that youâre searching for, that you want to include in a pattern</li> +</ul> +<p>Pattern</p><ul> +<li>regular PostgreSQL pattern match expression of symbols and operators that you want to match across rows</li> +</ul> +<p>Pattern match</p><ul> +<li>rows that result from a pattern match expression of symbols</li> +<li>can be multiple matches per partition</li> +</ul> +<p><a class="anchor" id="literature"></a></p><dl class="section user"><dt>Literature</dt><dd></dd></dl> +<p>[1] PostgreSQL basic statements/assignment operator, <a href="http://www.postgresql.org/docs/8.2/static/plpgsql-statements.html";>http://www.postgresql.org/docs/8.2/static/plpgsql-statements.html</a></p> +<p>[2] PostgreSQL pattern matching, <a href="http://www.postgresql.org/docs/current/static/functions-matching.html";>http://www.postgresql.org/docs/current/static/functions-matching.html</a></p> +<p>[3] PostgreSQL aggregate functions, <a href="http://www.postgresql.org/docs/8.2/static/tutorial-agg.html";>http://www.postgresql.org/docs/8.2/static/tutorial-agg.html</a></p> +<p>[4] PostgreSQL window functions, <a href="http://www.postgresql.org/docs/8.4/static/tutorial-window.html";>http://www.postgresql.org/docs/8.4/static/tutorial-window.html</a> </p> +</div><!-- contents --> +</div><!-- doc-content --> +<!-- start footer part --> +<div id="nav-path" class="navpath"><!-- id is needed for treeview function! --> + <ul> + <li class="footer">Generated on Tue May 16 2017 13:24:39 for MADlib by + <a href="http://www.doxygen.org/index.html";> + <img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.13 </li> + </ul> +</div> +</body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/b5b51c69/docs/v1.11/group__grp__pca.html ---------------------------------------------------------------------- diff --git a/docs/v1.11/group__grp__pca.html b/docs/v1.11/group__grp__pca.html new file mode 100644 index 0000000..126ef28 --- /dev/null +++ b/docs/v1.11/group__grp__pca.html @@ -0,0 +1,136 @@ +<!-- HTML header for doxygen 1.8.4--> +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> +<html xmlns="http://www.w3.org/1999/xhtml";> +<head> +<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/> +<meta http-equiv="X-UA-Compatible" content="IE=9"/> +<meta name="generator" content="Doxygen 1.8.13"/> +<meta name="keywords" content="madlib,postgres,greenplum,machine learning,data mining,deep learning,ensemble methods,data science,market basket analysis,affinity analysis,pca,lda,regression,elastic net,huber white,proportional hazards,k-means,latent dirichlet allocation,bayes,support vector machines,svm"/> +<title>MADlib: Dimensionality Reduction</title> +<link href="tabs.css" rel="stylesheet" type="text/css"/> +<script type="text/javascript" src="jquery.js"></script> +<script type="text/javascript" src="dynsections.js"></script> +<link href="navtree.css" rel="stylesheet" type="text/css"/> +<script type="text/javascript" src="resize.js"></script> +<script type="text/javascript" src="navtreedata.js"></script> +<script type="text/javascript" src="navtree.js"></script> +<script type="text/javascript"> + $(document).ready(initResizable); +</script> +<link href="search/search.css" rel="stylesheet" type="text/css"/> +<script type="text/javascript" src="search/searchdata.js"></script> +<script type="text/javascript" src="search/search.js"></script> +<script type="text/javascript"> + $(document).ready(function() { init_search(); }); +</script> +<!-- hack in the navigation tree --> +<script type="text/javascript" src="eigen_navtree_hacks.js"></script> +<link href="doxygen.css" rel="stylesheet" type="text/css" /> +<link href="madlib_extra.css" rel="stylesheet" type="text/css"/> +<!-- google analytics --> +<script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + ga('create', 'UA-45382226-1', 'madlib.incubator.apache.org'); + ga('send', 'pageview'); +</script> +</head> +<body> +<div id="top"><!-- do not remove this div, it is closed by doxygen! --> +<div id="titlearea"> +<table cellspacing="0" cellpadding="0"> + <tbody> + <tr style="height: 56px;"> + <td id="projectlogo"><a href="http://madlib.incubator.apache.org";><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td> + <td style="padding-left: 0.5em;"> + <div id="projectname"> + <span id="projectnumber">1.11</span> + </div> + <div id="projectbrief">User Documentation for MADlib</div> + </td> + <td> <div id="MSearchBox" class="MSearchBoxInactive"> + <span class="left"> + <img id="MSearchSelect" src="search/mag_sel.png" + onmouseover="return searchBox.OnSearchSelectShow()" + onmouseout="return searchBox.OnSearchSelectHide()" + alt=""/> + <input type="text" id="MSearchField" value="Search" accesskey="S" + onfocus="searchBox.OnSearchFieldFocus(true)" + onblur="searchBox.OnSearchFieldFocus(false)" + onkeyup="searchBox.OnSearchFieldChange(event)"/> + </span><span class="right"> + <a id="MSearchClose" href="javascript:searchBox.CloseResultsWindow()"><img id="MSearchCloseImg" border="0" src="search/close.png" alt=""/></a> + </span> + </div> +</td> + </tr> + </tbody> +</table> +</div> +<!-- end header part --> +<!-- Generated by Doxygen 1.8.13 --> +<script type="text/javascript"> +var searchBox = new SearchBox("searchBox", "search",false,'Search'); +</script> +</div><!-- top --> +<div id="side-nav" class="ui-resizable side-nav-resizable"> + <div id="nav-tree"> + <div id="nav-tree-contents"> + <div id="nav-sync" class="sync"></div> + </div> + </div> + <div id="splitbar" style="-moz-user-select:none;" + class="ui-resizable-handle"> + </div> +</div> +<script type="text/javascript"> +$(document).ready(function(){initNavTree('group__grp__pca.html','');}); +</script> +<div id="doc-content"> +<!-- window showing the filter options --> +<div id="MSearchSelectWindow" + onmouseover="return searchBox.OnSearchSelectShow()" + onmouseout="return searchBox.OnSearchSelectHide()" + onkeydown="return searchBox.OnSearchSelectKey(event)"> +</div> + +<!-- iframe showing the search results (closed by default) --> +<div id="MSearchResultsWindow"> +<iframe src="javascript:void(0)" frameborder="0" + name="MSearchResults" id="MSearchResults"> +</iframe> +</div> + +<div class="header"> + <div class="summary"> +<a href="#groups">Modules</a> </div> + <div class="headertitle"> +<div class="title">Dimensionality Reduction<div class="ingroups"><a class="el" href="group__grp__datatrans.html">Data Types and Transformations</a></div></div> </div> +</div><!--header--> +<div class="contents"> +<a name="details" id="details"></a><h2 class="groupheader">Detailed Description</h2> +<p>A collection of methods for dimensionality reduction. </p> +<table class="memberdecls"> +<tr class="heading"><td colspan="2"><h2 class="groupheader"><a name="groups"></a> +Modules</h2></td></tr> +<tr class="memitem:group__grp__pca__train"><td class="memItemLeft" align="right" valign="top"> </td><td class="memItemRight" valign="bottom"><a class="el" href="group__grp__pca__train.html">Principal Component Analysis</a></td></tr> +<tr class="memdesc:group__grp__pca__train"><td class="mdescLeft"> </td><td class="mdescRight">Produces a model that transforms a number of (possibly) correlated variables into a (smaller) number of uncorrelated variables called principal components. <br /></td></tr> +<tr class="separator:"><td class="memSeparator" colspan="2"> </td></tr> +<tr class="memitem:group__grp__pca__project"><td class="memItemLeft" align="right" valign="top"> </td><td class="memItemRight" valign="bottom"><a class="el" href="group__grp__pca__project.html">Principal Component Projection</a></td></tr> +<tr class="memdesc:group__grp__pca__project"><td class="mdescLeft"> </td><td class="mdescRight">Projects a higher dimensional data point to a lower dimensional subspace spanned by principal components learned through the PCA training procedure. <br /></td></tr> +<tr class="separator:"><td class="memSeparator" colspan="2"> </td></tr> +</table> +</div><!-- contents --> +</div><!-- doc-content --> +<!-- start footer part --> +<div id="nav-path" class="navpath"><!-- id is needed for treeview function! --> + <ul> + <li class="footer">Generated on Tue May 16 2017 13:24:38 for MADlib by + <a href="http://www.doxygen.org/index.html";> + <img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.13 </li> + </ul> +</div> +</body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/b5b51c69/docs/v1.11/group__grp__pca.js ---------------------------------------------------------------------- diff --git a/docs/v1.11/group__grp__pca.js b/docs/v1.11/group__grp__pca.js new file mode 100644 index 0000000..2863cf8 --- /dev/null +++ b/docs/v1.11/group__grp__pca.js @@ -0,0 +1,5 @@ +var group__grp__pca = +[ + [ "Principal Component Analysis", "group__grp__pca__train.html", null ], + [ "Principal Component Projection", "group__grp__pca__project.html", null ] +]; \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/b5b51c69/docs/v1.11/group__grp__pca__project.html ---------------------------------------------------------------------- diff --git a/docs/v1.11/group__grp__pca__project.html b/docs/v1.11/group__grp__pca__project.html new file mode 100644 index 0000000..50b3d51 --- /dev/null +++ b/docs/v1.11/group__grp__pca__project.html @@ -0,0 +1,493 @@ +<!-- HTML header for doxygen 1.8.4--> +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> +<html xmlns="http://www.w3.org/1999/xhtml";> +<head> +<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/> +<meta http-equiv="X-UA-Compatible" content="IE=9"/> +<meta name="generator" content="Doxygen 1.8.13"/> +<meta name="keywords" content="madlib,postgres,greenplum,machine learning,data mining,deep learning,ensemble methods,data science,market basket analysis,affinity analysis,pca,lda,regression,elastic net,huber white,proportional hazards,k-means,latent dirichlet allocation,bayes,support vector machines,svm"/> +<title>MADlib: Principal Component Projection</title> +<link href="tabs.css" rel="stylesheet" type="text/css"/> +<script type="text/javascript" src="jquery.js"></script> +<script type="text/javascript" src="dynsections.js"></script> +<link href="navtree.css" rel="stylesheet" type="text/css"/> +<script type="text/javascript" src="resize.js"></script> +<script type="text/javascript" src="navtreedata.js"></script> +<script type="text/javascript" src="navtree.js"></script> +<script type="text/javascript"> + $(document).ready(initResizable); +</script> +<link href="search/search.css" rel="stylesheet" type="text/css"/> +<script type="text/javascript" src="search/searchdata.js"></script> +<script type="text/javascript" src="search/search.js"></script> +<script type="text/javascript"> + $(document).ready(function() { init_search(); }); +</script> +<!-- hack in the navigation tree --> +<script type="text/javascript" src="eigen_navtree_hacks.js"></script> +<link href="doxygen.css" rel="stylesheet" type="text/css" /> +<link href="madlib_extra.css" rel="stylesheet" type="text/css"/> +<!-- google analytics --> +<script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + ga('create', 'UA-45382226-1', 'madlib.incubator.apache.org'); + ga('send', 'pageview'); +</script> +</head> +<body> +<div id="top"><!-- do not remove this div, it is closed by doxygen! --> +<div id="titlearea"> +<table cellspacing="0" cellpadding="0"> + <tbody> + <tr style="height: 56px;"> + <td id="projectlogo"><a href="http://madlib.incubator.apache.org";><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td> + <td style="padding-left: 0.5em;"> + <div id="projectname"> + <span id="projectnumber">1.11</span> + </div> + <div id="projectbrief">User Documentation for MADlib</div> + </td> + <td> <div id="MSearchBox" class="MSearchBoxInactive"> + <span class="left"> + <img id="MSearchSelect" src="search/mag_sel.png" + onmouseover="return searchBox.OnSearchSelectShow()" + onmouseout="return searchBox.OnSearchSelectHide()" + alt=""/> + <input type="text" id="MSearchField" value="Search" accesskey="S" + onfocus="searchBox.OnSearchFieldFocus(true)" + onblur="searchBox.OnSearchFieldFocus(false)" + onkeyup="searchBox.OnSearchFieldChange(event)"/> + </span><span class="right"> + <a id="MSearchClose" href="javascript:searchBox.CloseResultsWindow()"><img id="MSearchCloseImg" border="0" src="search/close.png" alt=""/></a> + </span> + </div> +</td> + </tr> + </tbody> +</table> +</div> +<!-- end header part --> +<!-- Generated by Doxygen 1.8.13 --> +<script type="text/javascript"> +var searchBox = new SearchBox("searchBox", "search",false,'Search'); +</script> +</div><!-- top --> +<div id="side-nav" class="ui-resizable side-nav-resizable"> + <div id="nav-tree"> + <div id="nav-tree-contents"> + <div id="nav-sync" class="sync"></div> + </div> + </div> + <div id="splitbar" style="-moz-user-select:none;" + class="ui-resizable-handle"> + </div> +</div> +<script type="text/javascript"> +$(document).ready(function(){initNavTree('group__grp__pca__project.html','');}); +</script> +<div id="doc-content"> +<!-- window showing the filter options --> +<div id="MSearchSelectWindow" + onmouseover="return searchBox.OnSearchSelectShow()" + onmouseout="return searchBox.OnSearchSelectHide()" + onkeydown="return searchBox.OnSearchSelectKey(event)"> +</div> + +<!-- iframe showing the search results (closed by default) --> +<div id="MSearchResultsWindow"> +<iframe src="javascript:void(0)" frameborder="0" + name="MSearchResults" id="MSearchResults"> +</iframe> +</div> + +<div class="header"> + <div class="headertitle"> +<div class="title">Principal Component Projection<div class="ingroups"><a class="el" href="group__grp__datatrans.html">Data Types and Transformations</a> » <a class="el" href="group__grp__pca.html">Dimensionality Reduction</a></div></div> </div> +</div><!--header--> +<div class="contents"> +<div class="toc"><b>Contents</b> <ul> +<li class="level1"> +<a href="#project">Projection Function</a> </li> +<li class="level1"> +<a href="#examples">Examples</a> </li> +<li class="level1"> +<a href="#notes">Notes</a> </li> +<li class="level1"> +<a href="#background_project">Technical Background</a> </li> +<li class="level1"> +<a href="#related">Related Topics</a> </li> +</ul> +</div><p>Principal component projection is a mathematical procedure that projects high dimensional data onto a lower dimensional space. This lower dimensional space is defined by the <img class="formulaInl" alt="$ k $" src="form_98.png"/> principal components with the highest variance in the training data.</p> +<p>More details on the mathematics of PCA can be found in <a class="el" href="group__grp__pca__train.html">Principal Component Analysis</a> and some details about principal component projection calculations can be found in the <a class="el" href="group__grp__pca__project.html#background_project">Technical Background</a>.</p> +<p><a class="anchor" id="project"></a></p><dl class="section user"><dt>Projection Function</dt><dd>The projection functions are slightly different for dense and sparse matrices. For dense matrices: <pre class="syntax"> +madlib.pca_project( source_table, + pc_table, + out_table, + row_id, + residual_table, + result_summary_table + ) +</pre> For sparse matrices: <pre class="syntax"> +madlib.pca_sparse_project( source_table, + pc_table, + out_table, + row_id, + col_id, -- Sparse matrices only + val_id, -- Sparse matrices only + row_dim, -- Sparse matrices only + col_dim, -- Sparse matrices only + residual_table, + result_summary_table + ) +</pre></dd></dl> +<p><b>Arguments</b> </p><dl class="arglist"> +<dt>source_table </dt> +<dd><p class="startdd">TEXT. Source table name. Identical to <a class="el" href="pca_8sql__in.html#a31abf88e67a446a4f789764aa2c61e85">pca_train</a>, the input data matrix should have <img class="formulaInl" alt="$ N $" src="form_220.png"/> rows and <img class="formulaInl" alt="$ M $" src="form_175.png"/> columns, where <img class="formulaInl" alt="$ N $" src="form_220.png"/> is the number of data points, and <img class="formulaInl" alt="$ M $" src="form_175.png"/> is the number of features for each data point.</p> +<p>The input table for <em> pca_project </em> is expected to be in the one of the two standard MADlib dense matrix formats, and the sparse input table for <em> pca_sparse_project </em> should be in the standard MADlib sparse matrix format. These formats are described in the documentation for <a class="el" href="group__grp__pca__train.html">Principal Component Analysis</a>.</p> +<p class="enddd"></p> +</dd> +<dt>pc_table </dt> +<dd><p class="startdd">TEXT. Table name for the table containing principal components. </p> +<p class="enddd"></p> +</dd> +<dt>out_table </dt> +<dd><p class="startdd">TEXT. Name of the table that will contain the low-dimensional representation of the input data.</p> +<p>The <em>out_table</em> encodes a dense matrix with the projection onto the principal components. The table has the following columns:</p> +<table class="output"> +<tr> +<th>row_id </th><td>Row id of the output matrix. </td></tr> +<tr> +<th>row_vec </th><td>A vector containing elements in the row of the matrix. </td></tr> +</table> +<p class="enddd"></p> +</dd> +<dt>row_id </dt> +<dd><p class="startdd">TEXT. Column name containing the row IDs in the input source table. The column should be of type INT (or a type that can be cast to INT) and should only contain values between 1 and <em>N</em>. For dense matrix format, it should contain all continguous integers from 1 to <em>N</em> describing the full matrix.</p> +<p class="enddd"></p> +</dd> +<dt>col_id </dt> +<dd><p class="startdd">TEXT. Column name containing the column IDs in sparse matrix representation. The column should be of type INT (or a type that can be cast to INT) and should only contain values between 1 and <em>M</em>. <em>This parameter applies to sparse matrices only.</em></p> +<p class="enddd"></p> +</dd> +<dt>val_id </dt> +<dd><p class="startdd">TEXT. Name of 'val_id' column in sparse matrix representation defining the values of the nonzero entries. <em>This parameter applies to sparse matrices only.</em></p> +<p class="enddd"></p> +</dd> +<dt>row_dim </dt> +<dd><p class="startdd">INTEGER. The actual number of rows in the matrix. That is, if the matrix was transformed into dense format, this is the number of rows it would have. <em>This parameter applies to sparse matrices only.</em></p> +<p class="enddd"></p> +</dd> +<dt>col_dim </dt> +<dd><p class="startdd">INTEGER. The actual number of columns in the matrix. That is, if the matrix was transformed into dense format, this is the number of columns it would have. <em>This parameter applies to sparse matrices only.</em></p> +<dl class="section note"><dt>Note</dt><dd>The parameters 'row_dim' and 'col_dim' could actually be inferred from the sparse matrix representation, so they will be removed in the future. For now they are maintained for backward compatability so you must enter them. Making 'row_dim' or 'col_dim' larger than the actual matrix has the effect of padding it with zeros, which is probably not useful.</dd></dl> +</dd> +<dt>residual_table (optional) </dt> +<dd><p class="startdd">TEXT, default: NULL. Name of the optional residual table.</p> +<p>The <em>residual_table</em> encodes a dense residual matrix. The table has the following columns:</p> +<table class="output"> +<tr> +<th>row_id </th><td>Row id of the output matrix. </td></tr> +<tr> +<th>row_vec </th><td>A vector containing elements in the row of the residual matrix. </td></tr> +</table> +<p class="enddd"></p> +</dd> +<dt>result_summary_table (optional) </dt> +<dd><p class="startdd">TEXT, default: NULL. Name of the optional summary table.</p> +<p class="enddd">The <em>result_summary_table</em> contains information about the performance time of the PCA projection. The table has the following columns: </p><table class="output"> +<tr> +<th>exec_time </th><td>Elapsed time (ms) for execution of the function. </td></tr> +<tr> +<th>residual_norm </th><td>Absolute error of the residuals. </td></tr> +<tr> +<th>relative_residual_norm </th><td>Relative error of the residuals. </td></tr> +</table> +</dd> +</dl> +<p><a class="anchor" id="examples"></a></p><dl class="section user"><dt>Examples</dt><dd><ol type="1"> +<li>View online help for the PCA projection function: <pre class="example"> +SELECT madlib.pca_project(); +</pre></li> +<li>Create sample data in dense matrix form: <pre class="example"> +DROP TABLE IF EXISTS mat; +CREATE TABLE mat (id integer, + row_vec double precision[] + ); +INSERT INTO mat VALUES +(1, '{1,2,3}'), +(2, '{2,1,2}'), +(3, '{3,2,1}'); +</pre></li> +<li>Run the PCA function for a specified number of principal components and view the results: <pre class="example"> +DROP TABLE IF EXISTS result_table, result_table_mean; +SELECT madlib.pca_train('mat', -- Source table + 'result_table', -- Output table + 'id', -- Row id of source table + 2); -- Number of principal components +SELECT * FROM result_table ORDER BY row_id; +</pre> <pre class="result"> + row_id | principal_components | std_dev | proportion +--------+--------------------------------------------------------------+-------------------+------------------- + 1 | {0.707106781186547,-6.93889390390723e-18,-0.707106781186548} | 1.41421356237309 | 0.857142857142244 + 2 | {0,1,0} | 0.577350269189626 | 0.142857142857041 +(2 rows) +</pre></li> +<li>Project the original data to a lower dimensional representation and view the result of the projection: <pre class="example"> +DROP TABLE IF EXISTS residual_table, result_summary_table, out_table; +SELECT madlib.pca_project( 'mat', + 'result_table', + 'out_table', + 'id', + 'residual_table', + 'result_summary_table' + ); +SELECT * FROM out_table ORDER BY row_id; +</pre> <pre class="result"> + row_id | row_vec +--------+-------------------------------------- + 1 | {-1.41421356237309,-0.33333333333} + 2 | {2.77555756157677e-17,0.66666666667} + 3 | {1.41421356237309,-0.33333333333} +(3 rows) +</pre> Check the error in the projection: <pre class="example"> +SELECT * FROM result_summary_table; +</pre> <pre class="result"> + exec_time | residual_norm | relative_residual_norm +---------------+-------------------+------------------------ + 331.792116165 | 5.89383520611e-16 | 9.68940539229e-17 +(1 row) +</pre> Check the residuals: <pre class="example"> +SELECT * FROM residual_table ORDER BY row_id; +</pre> <pre class="result"> + row_id | row_vec +--------+-------------------------------------------------------------------- + 1 | {-2.22044604925031e-16,-1.11022302462516e-16,3.33066907387547e-16} + 2 | {-1.12243865646685e-18,0,4.7381731349413e-17} + 3 | {2.22044604925031e-16,1.11022302462516e-16,-3.33066907387547e-16} +(3 rows) +</pre></li> +<li>Now we use grouping in dense form to learn different models for different groups. First, we create sample data in dense matrix form with a grouping column. Note we actually have different matrix sizes for the different groups, which is allowed for dense: <pre class="example"> +DROP TABLE IF EXISTS mat_group; +CREATE TABLE mat_group ( + id integer, + row_vec double precision[], + matrix_id integer +); +INSERT INTO mat_group VALUES +(1, '{1,2,3}', 1), +(2, '{2,1,2}', 1), +(3, '{3,2,1}', 1), +(4, '{1,2,3,4,5}', 2), +(5, '{2,5,2,4,1}', 2), +(6, '{5,4,3,2,1}', 2); +</pre></li> +<li>Run the PCA function with grouping for a specified proportion of variance and view the results: <pre class="example"> +DROP TABLE IF EXISTS result_table_group, result_table_group_mean; +SELECT madlib.pca_train('mat_group', -- Source table + 'result_table_group', -- Output table + 'id', -- Row id of source table + 0.8, -- Proportion of variance + 'matrix_id'); -- Grouping column +SELECT * FROM result_table_group ORDER BY matrix_id, row_id; +</pre> <pre class="result"> + row_id | principal_components | std_dev | proportion | matrix_id +--------+------------------------------------------------------------------------------------------------+-----------------+-------------------+----------- + 1 | {0.707106781186548,0,-0.707106781186547} | 1.4142135623731 | 0.857142857142245 | 1 + 1 | {-0.555378486712784,-0.388303582074091,0.0442457354870796,0.255566375612852,0.688115693174023} | 3.2315220311722 | 0.764102534485173 | 2 + 2 | {0.587384101786277,-0.485138064894743,0.311532046315153,-0.449458074050715,0.347212037159181} | 1.795531127192 | 0.235897465516047 | 2 +(3 rows) +</pre></li> +<li>Run the PCA projection on subsets of an input table based on grouping columns. Note that the parameter 'pc_table' used for projection must be generated in training using the same grouping columns. <pre class="example"> +DROP TABLE IF EXISTS mat_group_projected; +SELECT madlib.pca_project('mat_group', + 'result_table_group', + 'mat_group_projected', + 'id'); +SELECT * FROM mat_group_projected ORDER BY matrix_id, row_id; +</pre> <pre class="result"> + row_id | row_vec | matrix_id +--------+---------------------------------------+----------- + 1 | {1.4142135623731} | 1 + 2 | {7.40148683087139e-17} | 1 + 3 | {-1.4142135623731} | 1 + 4 | {-3.59290479201926,0.559694003674779} | 2 + 5 | {0.924092949098971,-2.00871628417505} | 2 + 6 | {2.66881184290186,1.44902228049511} | 2 +(6 rows) +</pre></li> +<li>Now let's look at sparse matrices. Create sample data in sparse matrix form: <pre class="example"> +DROP TABLE IF EXISTS mat_sparse; +CREATE TABLE mat_sparse ( + row_id integer, + col_id integer, + value double precision +); +INSERT INTO mat_sparse VALUES +(1, 1, 1.0), +(2, 2, 2.0), +(3, 3, 3.0), +(4, 4, 4.0), +(1, 5, 5.0), +(2, 4, 6.0), +(3, 2, 7.0), +(4, 3, 8.0); +</pre> As an aside, this is what the sparse matrix above looks like when put in dense form: <pre class="example"> +DROP TABLE IF EXISTS mat_dense; +SELECT madlib.matrix_densify('mat_sparse', + 'row=row_id, col=col_id, val=value', + 'mat_dense'); +SELECT * FROM mat_dense ORDER BY row_id; +</pre> <pre class="result"> + row_id | value +--------+------------- + 1 | {1,0,0,0,5} + 2 | {0,2,0,6,0} + 3 | {0,7,3,0,0} + 4 | {0,0,8,4,0} +(4 rows) +</pre></li> +<li>Run the PCA sparse function for a specified number of principal components and view the results: <pre class="example">DROP TABLE IF EXISTS result_table, result_table_mean; +SELECT madlib.pca_sparse_train( 'mat_sparse', -- Source table + 'result_table', -- Output table + 'row_id', -- Row id of source table + 'col_id', -- Column id of source table + 'value', -- Value of matrix at row_id, col_id + 4, -- Actual number of rows in the matrix + 5, -- Actual number of columns in the matrix + 3); -- Number of principal components +SELECT * FROM result_table ORDER BY row_id; +</pre> Result (with principal components truncated for readability): <pre class="result"> + row_id | principal_components | std_dev | proportion +--------+----------------------------------------------+------------------+------------------- + 1 | {-0.0876046030186158,-0.0968983772909994,... | 4.21362803829554 | 0.436590030617467 + 2 | {-0.0647272661608605,0.877639526308692,... | 3.68408023747461 | 0.333748701544697 + 3 | {-0.0780380267884855,0.177956517174911,... | 3.05606908060098 | 0.229661267837836 +(3 rows) +</pre></li> +<li>Project the original sparse data to low-dimensional representation: <pre class="example"> +DROP TABLE IF EXISTS mat_sparse_out; +SELECT madlib.pca_sparse_project( + 'mat_sparse', + 'result_table', + 'mat_sparse_out', + 'row_id', + 'col_id', + 'value', + 4, + 5 + ); +SELECT * FROM mat_sparse_out ORDER BY row_id; +</pre> <pre class="result"> + row_id | row_vec +--------+--------------------------------------------------------- + 1 | {4.66617015032369,-2.63552220635847,2.1865220849604} + 2 | {0.228360685652383,-1.21616275892926,-4.46864627611561} + 3 | {0.672067460100428,5.45249627172823,0.56445525585642} + 4 | {-5.5665982960765,-1.6008113064405,1.71766893529879} +(4 rows) +</pre></li> +<li>Now we use grouping in sparse form to learn different models for different groups. First, we create sample data in sparse matrix form with a grouping column: <pre class="example"> +DROP TABLE IF EXISTS mat_sparse_group; +CREATE TABLE mat_sparse_group ( + row_id integer, + col_id integer, + value double precision, + matrix_id integer); +INSERT INTO mat_sparse_group VALUES +(1, 1, 1.0, 1), +(2, 2, 2.0, 1), +(3, 3, 3.0, 1), +(4, 4, 4.0, 1), +(1, 5, 5.0, 1), +(2, 4, 6.0, 2), +(3, 2, 7.0, 2), +(4, 3, 8.0, 2); +</pre></li> +<li>Run the PCA function with grouping for a specified proportion of variance and view the results: <pre class="example"> +DROP TABLE IF EXISTS result_table_group, result_table_group_mean; +SELECT madlib.pca_sparse_train( 'mat_sparse_group', -- Source table + 'result_table_group', -- Output table + 'row_id', -- Row id of source table + 'col_id', -- Column id of source table + 'value', -- Value of matrix at row_id, col_id + 4, -- Actual number of rows in the matrix + 5, -- Actual number of columns in the matrix + 0.8, -- Proportion of variance + 'matrix_id'); +SELECT * FROM result_table_group ORDER BY matrix_id, row_id; +</pre> Result (with principal components truncated for readability): <pre class="result"> + row_id | principal_components | std_dev | proportion | matrix_id +--------+--------------------------------------------+------------------+-------------------+----------- + 1 | {-0.17805696611353,0.0681313257646983,... | 2.73659933165925 | 0.544652792875481 | 1 + 2 | {-0.0492086814863993,0.149371585357526,... | 2.06058314533194 | 0.308800210823714 | 1 + 1 | {0,-0.479486114660443,... | 4.40325305087975 | 0.520500333693473 | 2 + 2 | {0,0.689230898585949,... | 3.7435566458567 | 0.376220573442628 | 2 +(4 rows) +</pre></li> +<li>Projection in sparse format with grouping: <pre class="example"> +DROP TABLE IF EXISTS mat_sparse_group_projected; +SELECT madlib.pca_sparse_project( + 'mat_sparse_group', + 'result_table_group', + 'mat_sparse_group_projected', + 'row_id', + 'col_id', + 'value', + 4, + 5 + ); +SELECT * FROM mat_sparse_group_projected ORDER BY matrix_id, row_id; +</pre> <pre class="result"> + row_id | row_vec | matrix_id +--------+-----------------------------------------+----------- + 1 | {-4.00039298524261,-0.626820612715982} | 1 + 2 | {0.765350785238575,0.951348276645455} | 1 + 3 | {1.04951017256904,2.22388180170356} | 1 + 4 | {2.185532027435,-2.54840946563303} | 1 + 1 | {-0.627846810195469,-0.685031603549092} | 2 + 2 | {-1.64754249747757,-4.7662114622896} | 2 + 3 | {-3.98424961281857,4.13958468655255} | 2 + 4 | {6.25963892049161,1.31165837928614} | 2 +(8 rows) +</pre></li> +</ol> +</dd></dl> +<p><a class="anchor" id="notes"></a></p><dl class="section user"><dt>Notes</dt><dd><ul> +<li>This function is intended to operate on the principal component tables generated by <em> pca_train </em> or <em> pca_sparse_train</em>. The MADlib PCA functions generate a table containing the column-means in addition to a table containing the principal components. If this table is not found by the MADlib projection function, it will trigger an error. As long the principal component tables are created with MADlib functions, then the column-means table will be automatically found by the MADlib projection functions.</li> +<li>Because of the centering step in PCA projection (see "Technical Background"), sparse matrices almost always become dense during the projection process. Thus, this implementation automatically densifies sparse matrix input, and there should be no expected performance improvement in using sparse matrix input over dense matrix input.</li> +<li>Table names can be optionally schema qualified (current_schemas() is searched if a schema name is not provided) and all table and column names should follow case-sensitivity and quoting rules per the database. (For instance, 'mytable' and 'MyTable' both resolve to the same entity, i.e. 'mytable'. If mixed-case or multi-byte characters are desired for entity names then the string should be double-quoted; in this case the input would be '"MyTable"').</li> +<li>If the input table for pca_project (pca_sparse_project) contains grouping columns, the same grouping columns must be used in the training function used to generate the principal components too.</li> +</ul> +</dd></dl> +<p><a class="anchor" id="background_project"></a></p><dl class="section user"><dt>Technical Background</dt><dd></dd></dl> +<p>Given a table containing some principal components <img class="formulaInl" alt="$ \boldsymbol P $" src="form_231.png"/> and some input data <img class="formulaInl" alt="$ \boldsymbol X $" src="form_221.png"/>, the low-dimensional representation <img class="formulaInl" alt="$ {\boldsymbol X}' $" src="form_232.png"/> is computed as </p><p class="formulaDsp"> +<img class="formulaDsp" alt="\begin{align*} {\boldsymbol {\hat{X}}} & = {\boldsymbol X} - \vec{e} \hat{x}^T \\ {\boldsymbol X}' & = {\boldsymbol {\hat {X}}} {\boldsymbol P}. \end{align*}" src="form_233.png"/> +</p> +<p> where <img class="formulaInl" alt="$\hat{x} $" src="form_234.png"/> is the column means of <img class="formulaInl" alt="$ \boldsymbol X $" src="form_221.png"/> and <img class="formulaInl" alt="$ \vec{e} $" src="form_226.png"/> is the vector of all ones. This step is equivalent to centering the data around the origin.</p> +<p>The residual table <img class="formulaInl" alt="$ \boldsymbol R $" src="form_235.png"/> is a measure of how well the low-dimensional representation approximates the true input data, and is computed as </p><p class="formulaDsp"> +<img class="formulaDsp" alt="\[ {\boldsymbol R} = {\boldsymbol {\hat{X}}} - {\boldsymbol X}' {\boldsymbol P}^T. \]" src="form_236.png"/> +</p> +<p> A residual matrix with entries mostly close to zero indicates a good representation.</p> +<p>The residual norm <img class="formulaInl" alt="$ r $" src="form_237.png"/> is simply </p><p class="formulaDsp"> +<img class="formulaDsp" alt="\[ r = \|{\boldsymbol R}\|_F \]" src="form_238.png"/> +</p> +<p> where <img class="formulaInl" alt="$ \|\cdot\|_F $" src="form_239.png"/> is the Frobenius norm. The relative residual norm <img class="formulaInl" alt="$ r' $" src="form_240.png"/> is </p><p class="formulaDsp"> +<img class="formulaDsp" alt="\[ r' = \frac{ \|{\boldsymbol R}\|_F }{\|{\boldsymbol X}\|_F } \]" src="form_241.png"/> +</p> +<p><a class="anchor" id="related"></a></p><dl class="section user"><dt>Related Topics</dt><dd>File <a class="el" href="pca__project_8sql__in.html" title="Principal Component Analysis Projection. ">pca_project.sql_in</a> documenting the SQL functions</dd></dl> +<p><a class="el" href="group__grp__pca__train.html">Principal Component Analysis</a> </p> +</div><!-- contents --> +</div><!-- doc-content --> +<!-- start footer part --> +<div id="nav-path" class="navpath"><!-- id is needed for treeview function! --> + <ul> + <li class="footer">Generated on Tue May 16 2017 13:24:38 for MADlib by + <a href="http://www.doxygen.org/index.html";> + <img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.13 </li> + </ul> +</div> +</body> +</html>
