Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/eq_null_safe.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/eq_null_safe.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/eq_null_safe.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,76 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: %<=>%</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for %&lt;=&gt;% {SparkR}"><tr><td>%<=>% {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>%<=>%</h2> + +<h3>Description</h3> + +<p>Equality test that is safe for null values. +</p> + + +<h3>Usage</h3> + +<pre> +x %<=>% value + +## S4 method for signature 'Column' +x %<=>% value +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>x</code></td> +<td> +<p>a Column</p> +</td></tr> +<tr valign="top"><td><code>value</code></td> +<td> +<p>a value to compare</p> +</td></tr> +</table> + + +<h3>Details</h3> + +<p>Can be used, unlike standard equality operator, to perform null-safe joins. +Equivalent to Scala <code>Column.<=></code> and <code>Column.eqNullSafe</code>. +</p> + + +<h3>Note</h3> + +<p>%<=>% since 2.3.0 +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D df1 <- createDataFrame(data.frame( +##D x = c(1, NA, 3, NA), y = c(2, 6, 3, NA) +##D )) +##D +##D head(select(df1, df1$x == df1$y, df1$x %<=>% df1$y)) +##D +##D df2 <- createDataFrame(data.frame(y = c(3, NA))) +##D count(join(df1, df2, df1$y == df2$y)) +##D +##D count(join(df1, df2, df1$y %<=>% df2$y)) +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html>
Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/except.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/except.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/except.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,117 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: except</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for except {SparkR}"><tr><td>except {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>except</h2> + +<h3>Description</h3> + +<p>Return a new SparkDataFrame containing rows in this SparkDataFrame +but not in another SparkDataFrame. This is equivalent to <code>EXCEPT</code> in SQL. +</p> + + +<h3>Usage</h3> + +<pre> +except(x, y) + +## S4 method for signature 'SparkDataFrame,SparkDataFrame' +except(x, y) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>x</code></td> +<td> +<p>a SparkDataFrame.</p> +</td></tr> +<tr valign="top"><td><code>y</code></td> +<td> +<p>a SparkDataFrame.</p> +</td></tr> +</table> + + +<h3>Value</h3> + +<p>A SparkDataFrame containing the result of the except operation. +</p> + + +<h3>Note</h3> + +<p>except since 1.4.0 +</p> + + +<h3>See Also</h3> + +<p>Other SparkDataFrame functions: <code><a href="SparkDataFrame.html">SparkDataFrame-class</a></code>, +<code><a href="summarize.html">agg</a></code>, <code><a href="alias.html">alias</a></code>, +<code><a href="arrange.html">arrange</a></code>, <code><a href="as.data.frame.html">as.data.frame</a></code>, +<code><a href="attach.html">attach,SparkDataFrame-method</a></code>, +<code><a href="broadcast.html">broadcast</a></code>, <code><a href="cache.html">cache</a></code>, +<code><a href="checkpoint.html">checkpoint</a></code>, <code><a href="coalesce.html">coalesce</a></code>, +<code><a href="collect.html">collect</a></code>, <code><a href="columns.html">colnames</a></code>, +<code><a href="coltypes.html">coltypes</a></code>, +<code><a href="createOrReplaceTempView.html">createOrReplaceTempView</a></code>, +<code><a href="crossJoin.html">crossJoin</a></code>, <code><a href="cube.html">cube</a></code>, +<code><a href="dapplyCollect.html">dapplyCollect</a></code>, <code><a href="dapply.html">dapply</a></code>, +<code><a href="describe.html">describe</a></code>, <code><a href="dim.html">dim</a></code>, +<code><a href="distinct.html">distinct</a></code>, <code><a href="dropDuplicates.html">dropDuplicates</a></code>, +<code><a href="nafunctions.html">dropna</a></code>, <code><a href="drop.html">drop</a></code>, +<code><a href="dtypes.html">dtypes</a></code>, <code><a href="explain.html">explain</a></code>, +<code><a href="filter.html">filter</a></code>, <code><a href="first.html">first</a></code>, +<code><a href="gapplyCollect.html">gapplyCollect</a></code>, <code><a href="gapply.html">gapply</a></code>, +<code><a href="getNumPartitions.html">getNumPartitions</a></code>, <code><a href="groupBy.html">group_by</a></code>, +<code><a href="head.html">head</a></code>, <code><a href="hint.html">hint</a></code>, +<code><a href="histogram.html">histogram</a></code>, <code><a href="insertInto.html">insertInto</a></code>, +<code><a href="intersect.html">intersect</a></code>, <code><a href="isLocal.html">isLocal</a></code>, +<code><a href="isStreaming.html">isStreaming</a></code>, <code><a href="join.html">join</a></code>, +<code><a href="limit.html">limit</a></code>, <code><a href="localCheckpoint.html">localCheckpoint</a></code>, +<code><a href="merge.html">merge</a></code>, <code><a href="mutate.html">mutate</a></code>, +<code><a href="ncol.html">ncol</a></code>, <code><a href="nrow.html">nrow</a></code>, +<code><a href="persist.html">persist</a></code>, <code><a href="printSchema.html">printSchema</a></code>, +<code><a href="randomSplit.html">randomSplit</a></code>, <code><a href="rbind.html">rbind</a></code>, +<code><a href="registerTempTable-deprecated.html">registerTempTable</a></code>, <code><a href="rename.html">rename</a></code>, +<code><a href="repartition.html">repartition</a></code>, <code><a href="rollup.html">rollup</a></code>, +<code><a href="sample.html">sample</a></code>, <code><a href="saveAsTable.html">saveAsTable</a></code>, +<code><a href="schema.html">schema</a></code>, <code><a href="selectExpr.html">selectExpr</a></code>, +<code><a href="select.html">select</a></code>, <code><a href="showDF.html">showDF</a></code>, +<code><a href="show.html">show</a></code>, <code><a href="storageLevel.html">storageLevel</a></code>, +<code><a href="str.html">str</a></code>, <code><a href="subset.html">subset</a></code>, +<code><a href="summary.html">summary</a></code>, <code><a href="take.html">take</a></code>, +<code><a href="toJSON.html">toJSON</a></code>, <code><a href="unionByName.html">unionByName</a></code>, +<code><a href="union.html">union</a></code>, <code><a href="unpersist.html">unpersist</a></code>, +<code><a href="withColumn.html">withColumn</a></code>, <code><a href="withWatermark.html">withWatermark</a></code>, +<code><a href="with.html">with</a></code>, <code><a href="write.df.html">write.df</a></code>, +<code><a href="write.jdbc.html">write.jdbc</a></code>, <code><a href="write.json.html">write.json</a></code>, +<code><a href="write.orc.html">write.orc</a></code>, <code><a href="write.parquet.html">write.parquet</a></code>, +<code><a href="write.stream.html">write.stream</a></code>, <code><a href="write.text.html">write.text</a></code> +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D sparkR.session() +##D df1 <- read.json(path) +##D df2 <- read.json(path2) +##D exceptDF <- except(df, df2) +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/explain.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/explain.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/explain.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,125 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: Explain</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for explain {SparkR}"><tr><td>explain {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>Explain</h2> + +<h3>Description</h3> + +<p>Print the logical and physical Catalyst plans to the console for debugging. +</p> + + +<h3>Usage</h3> + +<pre> +explain(x, ...) + +## S4 method for signature 'SparkDataFrame' +explain(x, extended = FALSE) + +## S4 method for signature 'StreamingQuery' +explain(x, extended = FALSE) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>x</code></td> +<td> +<p>a SparkDataFrame or a StreamingQuery.</p> +</td></tr> +<tr valign="top"><td><code>...</code></td> +<td> +<p>further arguments to be passed to or from other methods.</p> +</td></tr> +<tr valign="top"><td><code>extended</code></td> +<td> +<p>Logical. If extended is FALSE, prints only the physical plan.</p> +</td></tr> +</table> + + +<h3>Note</h3> + +<p>explain since 1.4.0 +</p> +<p>explain(StreamingQuery) since 2.2.0 +</p> + + +<h3>See Also</h3> + +<p>Other SparkDataFrame functions: <code><a href="SparkDataFrame.html">SparkDataFrame-class</a></code>, +<code><a href="summarize.html">agg</a></code>, <code><a href="alias.html">alias</a></code>, +<code><a href="arrange.html">arrange</a></code>, <code><a href="as.data.frame.html">as.data.frame</a></code>, +<code><a href="attach.html">attach,SparkDataFrame-method</a></code>, +<code><a href="broadcast.html">broadcast</a></code>, <code><a href="cache.html">cache</a></code>, +<code><a href="checkpoint.html">checkpoint</a></code>, <code><a href="coalesce.html">coalesce</a></code>, +<code><a href="collect.html">collect</a></code>, <code><a href="columns.html">colnames</a></code>, +<code><a href="coltypes.html">coltypes</a></code>, +<code><a href="createOrReplaceTempView.html">createOrReplaceTempView</a></code>, +<code><a href="crossJoin.html">crossJoin</a></code>, <code><a href="cube.html">cube</a></code>, +<code><a href="dapplyCollect.html">dapplyCollect</a></code>, <code><a href="dapply.html">dapply</a></code>, +<code><a href="describe.html">describe</a></code>, <code><a href="dim.html">dim</a></code>, +<code><a href="distinct.html">distinct</a></code>, <code><a href="dropDuplicates.html">dropDuplicates</a></code>, +<code><a href="nafunctions.html">dropna</a></code>, <code><a href="drop.html">drop</a></code>, +<code><a href="dtypes.html">dtypes</a></code>, <code><a href="except.html">except</a></code>, +<code><a href="filter.html">filter</a></code>, <code><a href="first.html">first</a></code>, +<code><a href="gapplyCollect.html">gapplyCollect</a></code>, <code><a href="gapply.html">gapply</a></code>, +<code><a href="getNumPartitions.html">getNumPartitions</a></code>, <code><a href="groupBy.html">group_by</a></code>, +<code><a href="head.html">head</a></code>, <code><a href="hint.html">hint</a></code>, +<code><a href="histogram.html">histogram</a></code>, <code><a href="insertInto.html">insertInto</a></code>, +<code><a href="intersect.html">intersect</a></code>, <code><a href="isLocal.html">isLocal</a></code>, +<code><a href="isStreaming.html">isStreaming</a></code>, <code><a href="join.html">join</a></code>, +<code><a href="limit.html">limit</a></code>, <code><a href="localCheckpoint.html">localCheckpoint</a></code>, +<code><a href="merge.html">merge</a></code>, <code><a href="mutate.html">mutate</a></code>, +<code><a href="ncol.html">ncol</a></code>, <code><a href="nrow.html">nrow</a></code>, +<code><a href="persist.html">persist</a></code>, <code><a href="printSchema.html">printSchema</a></code>, +<code><a href="randomSplit.html">randomSplit</a></code>, <code><a href="rbind.html">rbind</a></code>, +<code><a href="registerTempTable-deprecated.html">registerTempTable</a></code>, <code><a href="rename.html">rename</a></code>, +<code><a href="repartition.html">repartition</a></code>, <code><a href="rollup.html">rollup</a></code>, +<code><a href="sample.html">sample</a></code>, <code><a href="saveAsTable.html">saveAsTable</a></code>, +<code><a href="schema.html">schema</a></code>, <code><a href="selectExpr.html">selectExpr</a></code>, +<code><a href="select.html">select</a></code>, <code><a href="showDF.html">showDF</a></code>, +<code><a href="show.html">show</a></code>, <code><a href="storageLevel.html">storageLevel</a></code>, +<code><a href="str.html">str</a></code>, <code><a href="subset.html">subset</a></code>, +<code><a href="summary.html">summary</a></code>, <code><a href="take.html">take</a></code>, +<code><a href="toJSON.html">toJSON</a></code>, <code><a href="unionByName.html">unionByName</a></code>, +<code><a href="union.html">union</a></code>, <code><a href="unpersist.html">unpersist</a></code>, +<code><a href="withColumn.html">withColumn</a></code>, <code><a href="withWatermark.html">withWatermark</a></code>, +<code><a href="with.html">with</a></code>, <code><a href="write.df.html">write.df</a></code>, +<code><a href="write.jdbc.html">write.jdbc</a></code>, <code><a href="write.json.html">write.json</a></code>, +<code><a href="write.orc.html">write.orc</a></code>, <code><a href="write.parquet.html">write.parquet</a></code>, +<code><a href="write.stream.html">write.stream</a></code>, <code><a href="write.text.html">write.text</a></code> +</p> +<p>Other StreamingQuery methods: <code><a href="awaitTermination.html">awaitTermination</a></code>, +<code><a href="isActive.html">isActive</a></code>, <code><a href="lastProgress.html">lastProgress</a></code>, +<code><a href="queryName.html">queryName</a></code>, <code><a href="status.html">status</a></code>, +<code><a href="stopQuery.html">stopQuery</a></code> +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D sparkR.session() +##D path <- "path/to/file.json" +##D df <- read.json(path) +##D explain(df, TRUE) +## End(Not run) +## Not run: explain(sq) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/filter.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/filter.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/filter.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,128 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: Filter</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for filter {SparkR}"><tr><td>filter {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>Filter</h2> + +<h3>Description</h3> + +<p>Filter the rows of a SparkDataFrame according to a given condition. +</p> + + +<h3>Usage</h3> + +<pre> +filter(x, condition) + +where(x, condition) + +## S4 method for signature 'SparkDataFrame,characterOrColumn' +filter(x, condition) + +## S4 method for signature 'SparkDataFrame,characterOrColumn' +where(x, condition) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>x</code></td> +<td> +<p>A SparkDataFrame to be sorted.</p> +</td></tr> +<tr valign="top"><td><code>condition</code></td> +<td> +<p>The condition to filter on. This may either be a Column expression +or a string containing a SQL statement</p> +</td></tr> +</table> + + +<h3>Value</h3> + +<p>A SparkDataFrame containing only the rows that meet the condition. +</p> + + +<h3>Note</h3> + +<p>filter since 1.4.0 +</p> +<p>where since 1.4.0 +</p> + + +<h3>See Also</h3> + +<p>Other SparkDataFrame functions: <code><a href="SparkDataFrame.html">SparkDataFrame-class</a></code>, +<code><a href="summarize.html">agg</a></code>, <code><a href="alias.html">alias</a></code>, +<code><a href="arrange.html">arrange</a></code>, <code><a href="as.data.frame.html">as.data.frame</a></code>, +<code><a href="attach.html">attach,SparkDataFrame-method</a></code>, +<code><a href="broadcast.html">broadcast</a></code>, <code><a href="cache.html">cache</a></code>, +<code><a href="checkpoint.html">checkpoint</a></code>, <code><a href="coalesce.html">coalesce</a></code>, +<code><a href="collect.html">collect</a></code>, <code><a href="columns.html">colnames</a></code>, +<code><a href="coltypes.html">coltypes</a></code>, +<code><a href="createOrReplaceTempView.html">createOrReplaceTempView</a></code>, +<code><a href="crossJoin.html">crossJoin</a></code>, <code><a href="cube.html">cube</a></code>, +<code><a href="dapplyCollect.html">dapplyCollect</a></code>, <code><a href="dapply.html">dapply</a></code>, +<code><a href="describe.html">describe</a></code>, <code><a href="dim.html">dim</a></code>, +<code><a href="distinct.html">distinct</a></code>, <code><a href="dropDuplicates.html">dropDuplicates</a></code>, +<code><a href="nafunctions.html">dropna</a></code>, <code><a href="drop.html">drop</a></code>, +<code><a href="dtypes.html">dtypes</a></code>, <code><a href="except.html">except</a></code>, +<code><a href="explain.html">explain</a></code>, <code><a href="first.html">first</a></code>, +<code><a href="gapplyCollect.html">gapplyCollect</a></code>, <code><a href="gapply.html">gapply</a></code>, +<code><a href="getNumPartitions.html">getNumPartitions</a></code>, <code><a href="groupBy.html">group_by</a></code>, +<code><a href="head.html">head</a></code>, <code><a href="hint.html">hint</a></code>, +<code><a href="histogram.html">histogram</a></code>, <code><a href="insertInto.html">insertInto</a></code>, +<code><a href="intersect.html">intersect</a></code>, <code><a href="isLocal.html">isLocal</a></code>, +<code><a href="isStreaming.html">isStreaming</a></code>, <code><a href="join.html">join</a></code>, +<code><a href="limit.html">limit</a></code>, <code><a href="localCheckpoint.html">localCheckpoint</a></code>, +<code><a href="merge.html">merge</a></code>, <code><a href="mutate.html">mutate</a></code>, +<code><a href="ncol.html">ncol</a></code>, <code><a href="nrow.html">nrow</a></code>, +<code><a href="persist.html">persist</a></code>, <code><a href="printSchema.html">printSchema</a></code>, +<code><a href="randomSplit.html">randomSplit</a></code>, <code><a href="rbind.html">rbind</a></code>, +<code><a href="registerTempTable-deprecated.html">registerTempTable</a></code>, <code><a href="rename.html">rename</a></code>, +<code><a href="repartition.html">repartition</a></code>, <code><a href="rollup.html">rollup</a></code>, +<code><a href="sample.html">sample</a></code>, <code><a href="saveAsTable.html">saveAsTable</a></code>, +<code><a href="schema.html">schema</a></code>, <code><a href="selectExpr.html">selectExpr</a></code>, +<code><a href="select.html">select</a></code>, <code><a href="showDF.html">showDF</a></code>, +<code><a href="show.html">show</a></code>, <code><a href="storageLevel.html">storageLevel</a></code>, +<code><a href="str.html">str</a></code>, <code><a href="subset.html">subset</a></code>, +<code><a href="summary.html">summary</a></code>, <code><a href="take.html">take</a></code>, +<code><a href="toJSON.html">toJSON</a></code>, <code><a href="unionByName.html">unionByName</a></code>, +<code><a href="union.html">union</a></code>, <code><a href="unpersist.html">unpersist</a></code>, +<code><a href="withColumn.html">withColumn</a></code>, <code><a href="withWatermark.html">withWatermark</a></code>, +<code><a href="with.html">with</a></code>, <code><a href="write.df.html">write.df</a></code>, +<code><a href="write.jdbc.html">write.jdbc</a></code>, <code><a href="write.json.html">write.json</a></code>, +<code><a href="write.orc.html">write.orc</a></code>, <code><a href="write.parquet.html">write.parquet</a></code>, +<code><a href="write.stream.html">write.stream</a></code>, <code><a href="write.text.html">write.text</a></code> +</p> +<p>Other subsetting functions: <code><a href="select.html">select</a></code>, +<code><a href="subset.html">subset</a></code> +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D sparkR.session() +##D path <- "path/to/file.json" +##D df <- read.json(path) +##D filter(df, "col1 > 0") +##D filter(df, df$col2 != "abcdefg") +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/first.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/first.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/first.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,138 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: Return the first row of a SparkDataFrame</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for first {SparkR}"><tr><td>first {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>Return the first row of a SparkDataFrame</h2> + +<h3>Description</h3> + +<p>Return the first row of a SparkDataFrame +</p> +<p>Aggregate function: returns the first value in a group. +</p> + + +<h3>Usage</h3> + +<pre> +first(x, ...) + +## S4 method for signature 'SparkDataFrame' +first(x) + +## S4 method for signature 'characterOrColumn' +first(x, na.rm = FALSE) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>x</code></td> +<td> +<p>a SparkDataFrame or a column used in aggregation function.</p> +</td></tr> +<tr valign="top"><td><code>...</code></td> +<td> +<p>further arguments to be passed to or from other methods.</p> +</td></tr> +<tr valign="top"><td><code>na.rm</code></td> +<td> +<p>a logical value indicating whether NA values should be stripped +before the computation proceeds.</p> +</td></tr> +</table> + + +<h3>Details</h3> + +<p>The function by default returns the first values it sees. It will return the first non-missing +value it sees when na.rm is set to true. If all values are missing, then NA is returned. +</p> + + +<h3>Note</h3> + +<p>first(SparkDataFrame) since 1.4.0 +</p> +<p>first(characterOrColumn) since 1.4.0 +</p> + + +<h3>See Also</h3> + +<p>Other SparkDataFrame functions: <code><a href="SparkDataFrame.html">SparkDataFrame-class</a></code>, +<code><a href="summarize.html">agg</a></code>, <code><a href="alias.html">alias</a></code>, +<code><a href="arrange.html">arrange</a></code>, <code><a href="as.data.frame.html">as.data.frame</a></code>, +<code><a href="attach.html">attach,SparkDataFrame-method</a></code>, +<code><a href="broadcast.html">broadcast</a></code>, <code><a href="cache.html">cache</a></code>, +<code><a href="checkpoint.html">checkpoint</a></code>, <code><a href="coalesce.html">coalesce</a></code>, +<code><a href="collect.html">collect</a></code>, <code><a href="columns.html">colnames</a></code>, +<code><a href="coltypes.html">coltypes</a></code>, +<code><a href="createOrReplaceTempView.html">createOrReplaceTempView</a></code>, +<code><a href="crossJoin.html">crossJoin</a></code>, <code><a href="cube.html">cube</a></code>, +<code><a href="dapplyCollect.html">dapplyCollect</a></code>, <code><a href="dapply.html">dapply</a></code>, +<code><a href="describe.html">describe</a></code>, <code><a href="dim.html">dim</a></code>, +<code><a href="distinct.html">distinct</a></code>, <code><a href="dropDuplicates.html">dropDuplicates</a></code>, +<code><a href="nafunctions.html">dropna</a></code>, <code><a href="drop.html">drop</a></code>, +<code><a href="dtypes.html">dtypes</a></code>, <code><a href="except.html">except</a></code>, +<code><a href="explain.html">explain</a></code>, <code><a href="filter.html">filter</a></code>, +<code><a href="gapplyCollect.html">gapplyCollect</a></code>, <code><a href="gapply.html">gapply</a></code>, +<code><a href="getNumPartitions.html">getNumPartitions</a></code>, <code><a href="groupBy.html">group_by</a></code>, +<code><a href="head.html">head</a></code>, <code><a href="hint.html">hint</a></code>, +<code><a href="histogram.html">histogram</a></code>, <code><a href="insertInto.html">insertInto</a></code>, +<code><a href="intersect.html">intersect</a></code>, <code><a href="isLocal.html">isLocal</a></code>, +<code><a href="isStreaming.html">isStreaming</a></code>, <code><a href="join.html">join</a></code>, +<code><a href="limit.html">limit</a></code>, <code><a href="localCheckpoint.html">localCheckpoint</a></code>, +<code><a href="merge.html">merge</a></code>, <code><a href="mutate.html">mutate</a></code>, +<code><a href="ncol.html">ncol</a></code>, <code><a href="nrow.html">nrow</a></code>, +<code><a href="persist.html">persist</a></code>, <code><a href="printSchema.html">printSchema</a></code>, +<code><a href="randomSplit.html">randomSplit</a></code>, <code><a href="rbind.html">rbind</a></code>, +<code><a href="registerTempTable-deprecated.html">registerTempTable</a></code>, <code><a href="rename.html">rename</a></code>, +<code><a href="repartition.html">repartition</a></code>, <code><a href="rollup.html">rollup</a></code>, +<code><a href="sample.html">sample</a></code>, <code><a href="saveAsTable.html">saveAsTable</a></code>, +<code><a href="schema.html">schema</a></code>, <code><a href="selectExpr.html">selectExpr</a></code>, +<code><a href="select.html">select</a></code>, <code><a href="showDF.html">showDF</a></code>, +<code><a href="show.html">show</a></code>, <code><a href="storageLevel.html">storageLevel</a></code>, +<code><a href="str.html">str</a></code>, <code><a href="subset.html">subset</a></code>, +<code><a href="summary.html">summary</a></code>, <code><a href="take.html">take</a></code>, +<code><a href="toJSON.html">toJSON</a></code>, <code><a href="unionByName.html">unionByName</a></code>, +<code><a href="union.html">union</a></code>, <code><a href="unpersist.html">unpersist</a></code>, +<code><a href="withColumn.html">withColumn</a></code>, <code><a href="withWatermark.html">withWatermark</a></code>, +<code><a href="with.html">with</a></code>, <code><a href="write.df.html">write.df</a></code>, +<code><a href="write.jdbc.html">write.jdbc</a></code>, <code><a href="write.json.html">write.json</a></code>, +<code><a href="write.orc.html">write.orc</a></code>, <code><a href="write.parquet.html">write.parquet</a></code>, +<code><a href="write.stream.html">write.stream</a></code>, <code><a href="write.text.html">write.text</a></code> +</p> +<p>Other aggregate functions: <code><a href="avg.html">avg</a></code>, +<code><a href="column_aggregate_functions.html">column_aggregate_functions</a></code>, +<code><a href="corr.html">corr</a></code>, <code><a href="count.html">count</a></code>, +<code><a href="cov.html">cov</a></code>, <code><a href="last.html">last</a></code> +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D sparkR.session() +##D path <- "path/to/file.json" +##D df <- read.json(path) +##D first(df) +## End(Not run) +## Not run: +##D first(df$c) +##D first(df$c, TRUE) +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/fitted.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/fitted.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/fitted.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,74 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: Get fitted result from a k-means model</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for fitted {SparkR}"><tr><td>fitted {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>Get fitted result from a k-means model</h2> + +<h3>Description</h3> + +<p>Get fitted result from a k-means model, similarly to R's fitted(). +Note: A saved-loaded model does not support this method. +</p> + + +<h3>Usage</h3> + +<pre> +fitted(object, ...) + +## S4 method for signature 'KMeansModel' +fitted(object, method = c("centers", "classes")) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>object</code></td> +<td> +<p>a fitted k-means model.</p> +</td></tr> +<tr valign="top"><td><code>...</code></td> +<td> +<p>additional argument(s) passed to the method.</p> +</td></tr> +<tr valign="top"><td><code>method</code></td> +<td> +<p>type of fitted results, <code>"centers"</code> for cluster centers +or <code>"classes"</code> for assigned classes.</p> +</td></tr> +</table> + + +<h3>Value</h3> + +<p><code>fitted</code> returns a SparkDataFrame containing fitted values. +</p> + + +<h3>Note</h3> + +<p>fitted since 2.0.0 +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D model <- spark.kmeans(trainingData, ~ ., 2) +##D fitted.model <- fitted(model) +##D showDF(fitted.model) +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/freqItems.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/freqItems.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/freqItems.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,80 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: Finding frequent items for columns, possibly with false...</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for freqItems {SparkR}"><tr><td>freqItems {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>Finding frequent items for columns, possibly with false positives</h2> + +<h3>Description</h3> + +<p>Finding frequent items for columns, possibly with false positives. +Using the frequent element count algorithm described in +<a href="http://dx.doi.org/10.1145/762471.762473">http://dx.doi.org/10.1145/762471.762473</a>, proposed by Karp, Schenker, and Papadimitriou. +</p> + + +<h3>Usage</h3> + +<pre> +## S4 method for signature 'SparkDataFrame,character' +freqItems(x, cols, support = 0.01) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>x</code></td> +<td> +<p>A SparkDataFrame.</p> +</td></tr> +<tr valign="top"><td><code>cols</code></td> +<td> +<p>A vector column names to search frequent items in.</p> +</td></tr> +<tr valign="top"><td><code>support</code></td> +<td> +<p>(Optional) The minimum frequency for an item to be considered <code>frequent</code>. +Should be greater than 1e-4. Default support = 0.01.</p> +</td></tr> +</table> + + +<h3>Value</h3> + +<p>a local R data.frame with the frequent items in each column +</p> + + +<h3>Note</h3> + +<p>freqItems since 1.6.0 +</p> + + +<h3>See Also</h3> + +<p>Other stat functions: <code><a href="approxQuantile.html">approxQuantile</a></code>, +<code><a href="corr.html">corr</a></code>, <code><a href="cov.html">cov</a></code>, +<code><a href="crosstab.html">crosstab</a></code>, <code><a href="sampleBy.html">sampleBy</a></code> +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D df <- read.json("/path/to/file.json") +##D fi = freqItems(df, c("title", "gender")) +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/gapply.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/gapply.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/gapply.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,208 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: gapply</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for gapply {SparkR}"><tr><td>gapply {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>gapply</h2> + +<h3>Description</h3> + +<p>gapply +</p> +<p>Groups the SparkDataFrame using the specified columns and applies the R function to each +group. +</p> + + +<h3>Usage</h3> + +<pre> +gapply(x, ...) + +## S4 method for signature 'GroupedData' +gapply(x, func, schema) + +## S4 method for signature 'SparkDataFrame' +gapply(x, cols, func, schema) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>x</code></td> +<td> +<p>a SparkDataFrame or GroupedData.</p> +</td></tr> +<tr valign="top"><td><code>...</code></td> +<td> +<p>additional argument(s) passed to the method.</p> +</td></tr> +<tr valign="top"><td><code>func</code></td> +<td> +<p>a function to be applied to each group partition specified by grouping +column of the SparkDataFrame. The function <code>func</code> takes as argument +a key - grouping columns and a data frame - a local R data.frame. +The output of <code>func</code> is a local R data.frame.</p> +</td></tr> +<tr valign="top"><td><code>schema</code></td> +<td> +<p>the schema of the resulting SparkDataFrame after the function is applied. +The schema must match to output of <code>func</code>. It has to be defined for each +output column with preferred output column name and corresponding data type. +Since Spark 2.3, the DDL-formatted string is also supported for the schema.</p> +</td></tr> +<tr valign="top"><td><code>cols</code></td> +<td> +<p>grouping columns.</p> +</td></tr> +</table> + + +<h3>Value</h3> + +<p>A SparkDataFrame. +</p> + + +<h3>Note</h3> + +<p>gapply(GroupedData) since 2.0.0 +</p> +<p>gapply(SparkDataFrame) since 2.0.0 +</p> + + +<h3>See Also</h3> + +<p><a href="gapplyCollect.html">gapplyCollect</a> +</p> +<p>Other SparkDataFrame functions: <code><a href="SparkDataFrame.html">SparkDataFrame-class</a></code>, +<code><a href="summarize.html">agg</a></code>, <code><a href="alias.html">alias</a></code>, +<code><a href="arrange.html">arrange</a></code>, <code><a href="as.data.frame.html">as.data.frame</a></code>, +<code><a href="attach.html">attach,SparkDataFrame-method</a></code>, +<code><a href="broadcast.html">broadcast</a></code>, <code><a href="cache.html">cache</a></code>, +<code><a href="checkpoint.html">checkpoint</a></code>, <code><a href="coalesce.html">coalesce</a></code>, +<code><a href="collect.html">collect</a></code>, <code><a href="columns.html">colnames</a></code>, +<code><a href="coltypes.html">coltypes</a></code>, +<code><a href="createOrReplaceTempView.html">createOrReplaceTempView</a></code>, +<code><a href="crossJoin.html">crossJoin</a></code>, <code><a href="cube.html">cube</a></code>, +<code><a href="dapplyCollect.html">dapplyCollect</a></code>, <code><a href="dapply.html">dapply</a></code>, +<code><a href="describe.html">describe</a></code>, <code><a href="dim.html">dim</a></code>, +<code><a href="distinct.html">distinct</a></code>, <code><a href="dropDuplicates.html">dropDuplicates</a></code>, +<code><a href="nafunctions.html">dropna</a></code>, <code><a href="drop.html">drop</a></code>, +<code><a href="dtypes.html">dtypes</a></code>, <code><a href="except.html">except</a></code>, +<code><a href="explain.html">explain</a></code>, <code><a href="filter.html">filter</a></code>, +<code><a href="first.html">first</a></code>, <code><a href="gapplyCollect.html">gapplyCollect</a></code>, +<code><a href="getNumPartitions.html">getNumPartitions</a></code>, <code><a href="groupBy.html">group_by</a></code>, +<code><a href="head.html">head</a></code>, <code><a href="hint.html">hint</a></code>, +<code><a href="histogram.html">histogram</a></code>, <code><a href="insertInto.html">insertInto</a></code>, +<code><a href="intersect.html">intersect</a></code>, <code><a href="isLocal.html">isLocal</a></code>, +<code><a href="isStreaming.html">isStreaming</a></code>, <code><a href="join.html">join</a></code>, +<code><a href="limit.html">limit</a></code>, <code><a href="localCheckpoint.html">localCheckpoint</a></code>, +<code><a href="merge.html">merge</a></code>, <code><a href="mutate.html">mutate</a></code>, +<code><a href="ncol.html">ncol</a></code>, <code><a href="nrow.html">nrow</a></code>, +<code><a href="persist.html">persist</a></code>, <code><a href="printSchema.html">printSchema</a></code>, +<code><a href="randomSplit.html">randomSplit</a></code>, <code><a href="rbind.html">rbind</a></code>, +<code><a href="registerTempTable-deprecated.html">registerTempTable</a></code>, <code><a href="rename.html">rename</a></code>, +<code><a href="repartition.html">repartition</a></code>, <code><a href="rollup.html">rollup</a></code>, +<code><a href="sample.html">sample</a></code>, <code><a href="saveAsTable.html">saveAsTable</a></code>, +<code><a href="schema.html">schema</a></code>, <code><a href="selectExpr.html">selectExpr</a></code>, +<code><a href="select.html">select</a></code>, <code><a href="showDF.html">showDF</a></code>, +<code><a href="show.html">show</a></code>, <code><a href="storageLevel.html">storageLevel</a></code>, +<code><a href="str.html">str</a></code>, <code><a href="subset.html">subset</a></code>, +<code><a href="summary.html">summary</a></code>, <code><a href="take.html">take</a></code>, +<code><a href="toJSON.html">toJSON</a></code>, <code><a href="unionByName.html">unionByName</a></code>, +<code><a href="union.html">union</a></code>, <code><a href="unpersist.html">unpersist</a></code>, +<code><a href="withColumn.html">withColumn</a></code>, <code><a href="withWatermark.html">withWatermark</a></code>, +<code><a href="with.html">with</a></code>, <code><a href="write.df.html">write.df</a></code>, +<code><a href="write.jdbc.html">write.jdbc</a></code>, <code><a href="write.json.html">write.json</a></code>, +<code><a href="write.orc.html">write.orc</a></code>, <code><a href="write.parquet.html">write.parquet</a></code>, +<code><a href="write.stream.html">write.stream</a></code>, <code><a href="write.text.html">write.text</a></code> +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D Computes the arithmetic mean of the second column by grouping +##D on the first and third columns. Output the grouping values and the average. +##D +##D df <- createDataFrame ( +##D list(list(1L, 1, "1", 0.1), list(1L, 2, "1", 0.2), list(3L, 3, "3", 0.3)), +##D c("a", "b", "c", "d")) +##D +##D Here our output contains three columns, the key which is a combination of two +##D columns with data types integer and string and the mean which is a double. +##D schema <- structType(structField("a", "integer"), structField("c", "string"), +##D structField("avg", "double")) +##D result <- gapply( +##D df, +##D c("a", "c"), +##D function(key, x) { +##D y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) +##D }, schema) +##D +##D The schema also can be specified in a DDL-formatted string. +##D schema <- "a INT, c STRING, avg DOUBLE" +##D result <- gapply( +##D df, +##D c("a", "c"), +##D function(key, x) { +##D y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) +##D }, schema) +##D +##D We can also group the data and afterwards call gapply on GroupedData. +##D For Example: +##D gdf <- group_by(df, "a", "c") +##D result <- gapply( +##D gdf, +##D function(key, x) { +##D y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) +##D }, schema) +##D collect(result) +##D +##D Result +##D ------ +##D a c avg +##D 3 3 3.0 +##D 1 1 1.5 +##D +##D Fits linear models on iris dataset by grouping on the 'Species' column and +##D using 'Sepal_Length' as a target variable, 'Sepal_Width', 'Petal_Length' +##D and 'Petal_Width' as training features. +##D +##D df <- createDataFrame (iris) +##D schema <- structType(structField("(Intercept)", "double"), +##D structField("Sepal_Width", "double"),structField("Petal_Length", "double"), +##D structField("Petal_Width", "double")) +##D df1 <- gapply( +##D df, +##D df$"Species", +##D function(key, x) { +##D m <- suppressWarnings(lm(Sepal_Length ~ +##D Sepal_Width + Petal_Length + Petal_Width, x)) +##D data.frame(t(coef(m))) +##D }, schema) +##D collect(df1) +##D +##D Result +##D --------- +##D Model (Intercept) Sepal_Width Petal_Length Petal_Width +##D 1 0.699883 0.3303370 0.9455356 -0.1697527 +##D 2 1.895540 0.3868576 0.9083370 -0.6792238 +##D 3 2.351890 0.6548350 0.2375602 0.2521257 +##D +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/gapplyCollect.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/gapplyCollect.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/gapplyCollect.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,187 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: gapplyCollect</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for gapplyCollect {SparkR}"><tr><td>gapplyCollect {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>gapplyCollect</h2> + +<h3>Description</h3> + +<p>gapplyCollect +</p> +<p>Groups the SparkDataFrame using the specified columns, applies the R function to each +group and collects the result back to R as data.frame. +</p> + + +<h3>Usage</h3> + +<pre> +gapplyCollect(x, ...) + +## S4 method for signature 'GroupedData' +gapplyCollect(x, func) + +## S4 method for signature 'SparkDataFrame' +gapplyCollect(x, cols, func) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>x</code></td> +<td> +<p>a SparkDataFrame or GroupedData.</p> +</td></tr> +<tr valign="top"><td><code>...</code></td> +<td> +<p>additional argument(s) passed to the method.</p> +</td></tr> +<tr valign="top"><td><code>func</code></td> +<td> +<p>a function to be applied to each group partition specified by grouping +column of the SparkDataFrame. The function <code>func</code> takes as argument +a key - grouping columns and a data frame - a local R data.frame. +The output of <code>func</code> is a local R data.frame.</p> +</td></tr> +<tr valign="top"><td><code>cols</code></td> +<td> +<p>grouping columns.</p> +</td></tr> +</table> + + +<h3>Value</h3> + +<p>A data.frame. +</p> + + +<h3>Note</h3> + +<p>gapplyCollect(GroupedData) since 2.0.0 +</p> +<p>gapplyCollect(SparkDataFrame) since 2.0.0 +</p> + + +<h3>See Also</h3> + +<p><a href="gapply.html">gapply</a> +</p> +<p>Other SparkDataFrame functions: <code><a href="SparkDataFrame.html">SparkDataFrame-class</a></code>, +<code><a href="summarize.html">agg</a></code>, <code><a href="alias.html">alias</a></code>, +<code><a href="arrange.html">arrange</a></code>, <code><a href="as.data.frame.html">as.data.frame</a></code>, +<code><a href="attach.html">attach,SparkDataFrame-method</a></code>, +<code><a href="broadcast.html">broadcast</a></code>, <code><a href="cache.html">cache</a></code>, +<code><a href="checkpoint.html">checkpoint</a></code>, <code><a href="coalesce.html">coalesce</a></code>, +<code><a href="collect.html">collect</a></code>, <code><a href="columns.html">colnames</a></code>, +<code><a href="coltypes.html">coltypes</a></code>, +<code><a href="createOrReplaceTempView.html">createOrReplaceTempView</a></code>, +<code><a href="crossJoin.html">crossJoin</a></code>, <code><a href="cube.html">cube</a></code>, +<code><a href="dapplyCollect.html">dapplyCollect</a></code>, <code><a href="dapply.html">dapply</a></code>, +<code><a href="describe.html">describe</a></code>, <code><a href="dim.html">dim</a></code>, +<code><a href="distinct.html">distinct</a></code>, <code><a href="dropDuplicates.html">dropDuplicates</a></code>, +<code><a href="nafunctions.html">dropna</a></code>, <code><a href="drop.html">drop</a></code>, +<code><a href="dtypes.html">dtypes</a></code>, <code><a href="except.html">except</a></code>, +<code><a href="explain.html">explain</a></code>, <code><a href="filter.html">filter</a></code>, +<code><a href="first.html">first</a></code>, <code><a href="gapply.html">gapply</a></code>, +<code><a href="getNumPartitions.html">getNumPartitions</a></code>, <code><a href="groupBy.html">group_by</a></code>, +<code><a href="head.html">head</a></code>, <code><a href="hint.html">hint</a></code>, +<code><a href="histogram.html">histogram</a></code>, <code><a href="insertInto.html">insertInto</a></code>, +<code><a href="intersect.html">intersect</a></code>, <code><a href="isLocal.html">isLocal</a></code>, +<code><a href="isStreaming.html">isStreaming</a></code>, <code><a href="join.html">join</a></code>, +<code><a href="limit.html">limit</a></code>, <code><a href="localCheckpoint.html">localCheckpoint</a></code>, +<code><a href="merge.html">merge</a></code>, <code><a href="mutate.html">mutate</a></code>, +<code><a href="ncol.html">ncol</a></code>, <code><a href="nrow.html">nrow</a></code>, +<code><a href="persist.html">persist</a></code>, <code><a href="printSchema.html">printSchema</a></code>, +<code><a href="randomSplit.html">randomSplit</a></code>, <code><a href="rbind.html">rbind</a></code>, +<code><a href="registerTempTable-deprecated.html">registerTempTable</a></code>, <code><a href="rename.html">rename</a></code>, +<code><a href="repartition.html">repartition</a></code>, <code><a href="rollup.html">rollup</a></code>, +<code><a href="sample.html">sample</a></code>, <code><a href="saveAsTable.html">saveAsTable</a></code>, +<code><a href="schema.html">schema</a></code>, <code><a href="selectExpr.html">selectExpr</a></code>, +<code><a href="select.html">select</a></code>, <code><a href="showDF.html">showDF</a></code>, +<code><a href="show.html">show</a></code>, <code><a href="storageLevel.html">storageLevel</a></code>, +<code><a href="str.html">str</a></code>, <code><a href="subset.html">subset</a></code>, +<code><a href="summary.html">summary</a></code>, <code><a href="take.html">take</a></code>, +<code><a href="toJSON.html">toJSON</a></code>, <code><a href="unionByName.html">unionByName</a></code>, +<code><a href="union.html">union</a></code>, <code><a href="unpersist.html">unpersist</a></code>, +<code><a href="withColumn.html">withColumn</a></code>, <code><a href="withWatermark.html">withWatermark</a></code>, +<code><a href="with.html">with</a></code>, <code><a href="write.df.html">write.df</a></code>, +<code><a href="write.jdbc.html">write.jdbc</a></code>, <code><a href="write.json.html">write.json</a></code>, +<code><a href="write.orc.html">write.orc</a></code>, <code><a href="write.parquet.html">write.parquet</a></code>, +<code><a href="write.stream.html">write.stream</a></code>, <code><a href="write.text.html">write.text</a></code> +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D Computes the arithmetic mean of the second column by grouping +##D on the first and third columns. Output the grouping values and the average. +##D +##D df <- createDataFrame ( +##D list(list(1L, 1, "1", 0.1), list(1L, 2, "1", 0.2), list(3L, 3, "3", 0.3)), +##D c("a", "b", "c", "d")) +##D +##D result <- gapplyCollect( +##D df, +##D c("a", "c"), +##D function(key, x) { +##D y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) +##D colnames(y) <- c("key_a", "key_c", "mean_b") +##D y +##D }) +##D +##D We can also group the data and afterwards call gapply on GroupedData. +##D For Example: +##D gdf <- group_by(df, "a", "c") +##D result <- gapplyCollect( +##D gdf, +##D function(key, x) { +##D y <- data.frame(key, mean(x$b), stringsAsFactors = FALSE) +##D colnames(y) <- c("key_a", "key_c", "mean_b") +##D y +##D }) +##D +##D Result +##D ------ +##D key_a key_c mean_b +##D 3 3 3.0 +##D 1 1 1.5 +##D +##D Fits linear models on iris dataset by grouping on the 'Species' column and +##D using 'Sepal_Length' as a target variable, 'Sepal_Width', 'Petal_Length' +##D and 'Petal_Width' as training features. +##D +##D df <- createDataFrame (iris) +##D result <- gapplyCollect( +##D df, +##D df$"Species", +##D function(key, x) { +##D m <- suppressWarnings(lm(Sepal_Length ~ +##D Sepal_Width + Petal_Length + Petal_Width, x)) +##D data.frame(t(coef(m))) +##D }) +##D +##D Result +##D --------- +##D Model X.Intercept. Sepal_Width Petal_Length Petal_Width +##D 1 0.699883 0.3303370 0.9455356 -0.1697527 +##D 2 1.895540 0.3868576 0.9083370 -0.6792238 +##D 3 2.351890 0.6548350 0.2375602 0.2521257 +##D +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/getLocalProperty.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/getLocalProperty.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/getLocalProperty.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,55 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: Get a local property set in this thread, or 'NULL' if it is...</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for getLocalProperty {SparkR}"><tr><td>getLocalProperty {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>Get a local property set in this thread, or <code>NULL</code> if it is missing. See +<code>setLocalProperty</code>.</h2> + +<h3>Description</h3> + +<p>Get a local property set in this thread, or <code>NULL</code> if it is missing. See +<code>setLocalProperty</code>. +</p> + + +<h3>Usage</h3> + +<pre> +getLocalProperty(key) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>key</code></td> +<td> +<p>The key for a local property.</p> +</td></tr> +</table> + + +<h3>Note</h3> + +<p>getLocalProperty since 2.3.0 +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D getLocalProperty("spark.scheduler.pool") +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/getNumPartitions.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/getNumPartitions.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/getNumPartitions.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,103 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: getNumPartitions</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for getNumPartitions {SparkR}"><tr><td>getNumPartitions {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>getNumPartitions</h2> + +<h3>Description</h3> + +<p>Return the number of partitions +</p> + + +<h3>Usage</h3> + +<pre> +## S4 method for signature 'SparkDataFrame' +getNumPartitions(x) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>x</code></td> +<td> +<p>A SparkDataFrame</p> +</td></tr> +</table> + + +<h3>Note</h3> + +<p>getNumPartitions since 2.1.1 +</p> + + +<h3>See Also</h3> + +<p>Other SparkDataFrame functions: <code><a href="SparkDataFrame.html">SparkDataFrame-class</a></code>, +<code><a href="summarize.html">agg</a></code>, <code><a href="alias.html">alias</a></code>, +<code><a href="arrange.html">arrange</a></code>, <code><a href="as.data.frame.html">as.data.frame</a></code>, +<code><a href="attach.html">attach,SparkDataFrame-method</a></code>, +<code><a href="broadcast.html">broadcast</a></code>, <code><a href="cache.html">cache</a></code>, +<code><a href="checkpoint.html">checkpoint</a></code>, <code><a href="coalesce.html">coalesce</a></code>, +<code><a href="collect.html">collect</a></code>, <code><a href="columns.html">colnames</a></code>, +<code><a href="coltypes.html">coltypes</a></code>, +<code><a href="createOrReplaceTempView.html">createOrReplaceTempView</a></code>, +<code><a href="crossJoin.html">crossJoin</a></code>, <code><a href="cube.html">cube</a></code>, +<code><a href="dapplyCollect.html">dapplyCollect</a></code>, <code><a href="dapply.html">dapply</a></code>, +<code><a href="describe.html">describe</a></code>, <code><a href="dim.html">dim</a></code>, +<code><a href="distinct.html">distinct</a></code>, <code><a href="dropDuplicates.html">dropDuplicates</a></code>, +<code><a href="nafunctions.html">dropna</a></code>, <code><a href="drop.html">drop</a></code>, +<code><a href="dtypes.html">dtypes</a></code>, <code><a href="except.html">except</a></code>, +<code><a href="explain.html">explain</a></code>, <code><a href="filter.html">filter</a></code>, +<code><a href="first.html">first</a></code>, <code><a href="gapplyCollect.html">gapplyCollect</a></code>, +<code><a href="gapply.html">gapply</a></code>, <code><a href="groupBy.html">group_by</a></code>, +<code><a href="head.html">head</a></code>, <code><a href="hint.html">hint</a></code>, +<code><a href="histogram.html">histogram</a></code>, <code><a href="insertInto.html">insertInto</a></code>, +<code><a href="intersect.html">intersect</a></code>, <code><a href="isLocal.html">isLocal</a></code>, +<code><a href="isStreaming.html">isStreaming</a></code>, <code><a href="join.html">join</a></code>, +<code><a href="limit.html">limit</a></code>, <code><a href="localCheckpoint.html">localCheckpoint</a></code>, +<code><a href="merge.html">merge</a></code>, <code><a href="mutate.html">mutate</a></code>, +<code><a href="ncol.html">ncol</a></code>, <code><a href="nrow.html">nrow</a></code>, +<code><a href="persist.html">persist</a></code>, <code><a href="printSchema.html">printSchema</a></code>, +<code><a href="randomSplit.html">randomSplit</a></code>, <code><a href="rbind.html">rbind</a></code>, +<code><a href="registerTempTable-deprecated.html">registerTempTable</a></code>, <code><a href="rename.html">rename</a></code>, +<code><a href="repartition.html">repartition</a></code>, <code><a href="rollup.html">rollup</a></code>, +<code><a href="sample.html">sample</a></code>, <code><a href="saveAsTable.html">saveAsTable</a></code>, +<code><a href="schema.html">schema</a></code>, <code><a href="selectExpr.html">selectExpr</a></code>, +<code><a href="select.html">select</a></code>, <code><a href="showDF.html">showDF</a></code>, +<code><a href="show.html">show</a></code>, <code><a href="storageLevel.html">storageLevel</a></code>, +<code><a href="str.html">str</a></code>, <code><a href="subset.html">subset</a></code>, +<code><a href="summary.html">summary</a></code>, <code><a href="take.html">take</a></code>, +<code><a href="toJSON.html">toJSON</a></code>, <code><a href="unionByName.html">unionByName</a></code>, +<code><a href="union.html">union</a></code>, <code><a href="unpersist.html">unpersist</a></code>, +<code><a href="withColumn.html">withColumn</a></code>, <code><a href="withWatermark.html">withWatermark</a></code>, +<code><a href="with.html">with</a></code>, <code><a href="write.df.html">write.df</a></code>, +<code><a href="write.jdbc.html">write.jdbc</a></code>, <code><a href="write.json.html">write.json</a></code>, +<code><a href="write.orc.html">write.orc</a></code>, <code><a href="write.parquet.html">write.parquet</a></code>, +<code><a href="write.stream.html">write.stream</a></code>, <code><a href="write.text.html">write.text</a></code> +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D sparkR.session() +##D df <- createDataFrame(cars, numPartitions = 2) +##D getNumPartitions(df) +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/glm.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/glm.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/glm.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,124 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: Generalized Linear Models (R-compliant)</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for glm,formula,ANY,SparkDataFrame-method {SparkR}"><tr><td>glm,formula,ANY,SparkDataFrame-method {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>Generalized Linear Models (R-compliant)</h2> + +<h3>Description</h3> + +<p>Fits a generalized linear model, similarly to R's glm(). +</p> + + +<h3>Usage</h3> + +<pre> +## S4 method for signature 'formula,ANY,SparkDataFrame' +glm(formula, family = gaussian, data, + epsilon = 1e-06, maxit = 25, weightCol = NULL, var.power = 0, + link.power = 1 - var.power, stringIndexerOrderType = c("frequencyDesc", + "frequencyAsc", "alphabetDesc", "alphabetAsc"), offsetCol = NULL) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>formula</code></td> +<td> +<p>a symbolic description of the model to be fitted. Currently only a few formula +operators are supported, including '~', '.', ':', '+', and '-'.</p> +</td></tr> +<tr valign="top"><td><code>family</code></td> +<td> +<p>a description of the error distribution and link function to be used in the model. +This can be a character string naming a family function, a family function or +the result of a call to a family function. Refer R family at +<a href="https://stat.ethz.ch/R-manual/R-devel/library/stats/html/family.html">https://stat.ethz.ch/R-manual/R-devel/library/stats/html/family.html</a>. +Currently these families are supported: <code>binomial</code>, <code>gaussian</code>, +<code>poisson</code>, <code>Gamma</code>, and <code>tweedie</code>.</p> +</td></tr> +<tr valign="top"><td><code>data</code></td> +<td> +<p>a SparkDataFrame or R's glm data for training.</p> +</td></tr> +<tr valign="top"><td><code>epsilon</code></td> +<td> +<p>positive convergence tolerance of iterations.</p> +</td></tr> +<tr valign="top"><td><code>maxit</code></td> +<td> +<p>integer giving the maximal number of IRLS iterations.</p> +</td></tr> +<tr valign="top"><td><code>weightCol</code></td> +<td> +<p>the weight column name. If this is not set or <code>NULL</code>, we treat all instance +weights as 1.0.</p> +</td></tr> +<tr valign="top"><td><code>var.power</code></td> +<td> +<p>the index of the power variance function in the Tweedie family.</p> +</td></tr> +<tr valign="top"><td><code>link.power</code></td> +<td> +<p>the index of the power link function in the Tweedie family.</p> +</td></tr> +<tr valign="top"><td><code>stringIndexerOrderType</code></td> +<td> +<p>how to order categories of a string feature column. This is used to +decide the base level of a string feature as the last category +after ordering is dropped when encoding strings. Supported options +are "frequencyDesc", "frequencyAsc", "alphabetDesc", and +"alphabetAsc". The default value is "frequencyDesc". When the +ordering is set to "alphabetDesc", this drops the same category +as R when encoding strings.</p> +</td></tr> +<tr valign="top"><td><code>offsetCol</code></td> +<td> +<p>the offset column name. If this is not set or empty, we treat all instance +offsets as 0.0. The feature specified as offset has a constant coefficient of +1.0.</p> +</td></tr> +</table> + + +<h3>Value</h3> + +<p><code>glm</code> returns a fitted generalized linear model. +</p> + + +<h3>Note</h3> + +<p>glm since 1.5.0 +</p> + + +<h3>See Also</h3> + +<p><a href="spark.glm.html">spark.glm</a> +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D sparkR.session() +##D t <- as.data.frame(Titanic) +##D df <- createDataFrame(t) +##D model <- glm(Freq ~ Sex + Age, df, family = "gaussian") +##D summary(model) +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/groupBy.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/groupBy.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/groupBy.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,126 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: GroupBy</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for group_by {SparkR}"><tr><td>group_by {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>GroupBy</h2> + +<h3>Description</h3> + +<p>Groups the SparkDataFrame using the specified columns, so we can run aggregation on them. +</p> + + +<h3>Usage</h3> + +<pre> +group_by(x, ...) + +groupBy(x, ...) + +## S4 method for signature 'SparkDataFrame' +groupBy(x, ...) + +## S4 method for signature 'SparkDataFrame' +group_by(x, ...) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>x</code></td> +<td> +<p>a SparkDataFrame.</p> +</td></tr> +<tr valign="top"><td><code>...</code></td> +<td> +<p>character name(s) or Column(s) to group on.</p> +</td></tr> +</table> + + +<h3>Value</h3> + +<p>A GroupedData. +</p> + + +<h3>Note</h3> + +<p>groupBy since 1.4.0 +</p> +<p>group_by since 1.4.0 +</p> + + +<h3>See Also</h3> + +<p><a href="summarize.html">agg</a>, <a href="cube.html">cube</a>, <a href="rollup.html">rollup</a> +</p> +<p>Other SparkDataFrame functions: <code><a href="SparkDataFrame.html">SparkDataFrame-class</a></code>, +<code><a href="summarize.html">agg</a></code>, <code><a href="alias.html">alias</a></code>, +<code><a href="arrange.html">arrange</a></code>, <code><a href="as.data.frame.html">as.data.frame</a></code>, +<code><a href="attach.html">attach,SparkDataFrame-method</a></code>, +<code><a href="broadcast.html">broadcast</a></code>, <code><a href="cache.html">cache</a></code>, +<code><a href="checkpoint.html">checkpoint</a></code>, <code><a href="coalesce.html">coalesce</a></code>, +<code><a href="collect.html">collect</a></code>, <code><a href="columns.html">colnames</a></code>, +<code><a href="coltypes.html">coltypes</a></code>, +<code><a href="createOrReplaceTempView.html">createOrReplaceTempView</a></code>, +<code><a href="crossJoin.html">crossJoin</a></code>, <code><a href="cube.html">cube</a></code>, +<code><a href="dapplyCollect.html">dapplyCollect</a></code>, <code><a href="dapply.html">dapply</a></code>, +<code><a href="describe.html">describe</a></code>, <code><a href="dim.html">dim</a></code>, +<code><a href="distinct.html">distinct</a></code>, <code><a href="dropDuplicates.html">dropDuplicates</a></code>, +<code><a href="nafunctions.html">dropna</a></code>, <code><a href="drop.html">drop</a></code>, +<code><a href="dtypes.html">dtypes</a></code>, <code><a href="except.html">except</a></code>, +<code><a href="explain.html">explain</a></code>, <code><a href="filter.html">filter</a></code>, +<code><a href="first.html">first</a></code>, <code><a href="gapplyCollect.html">gapplyCollect</a></code>, +<code><a href="gapply.html">gapply</a></code>, <code><a href="getNumPartitions.html">getNumPartitions</a></code>, +<code><a href="head.html">head</a></code>, <code><a href="hint.html">hint</a></code>, +<code><a href="histogram.html">histogram</a></code>, <code><a href="insertInto.html">insertInto</a></code>, +<code><a href="intersect.html">intersect</a></code>, <code><a href="isLocal.html">isLocal</a></code>, +<code><a href="isStreaming.html">isStreaming</a></code>, <code><a href="join.html">join</a></code>, +<code><a href="limit.html">limit</a></code>, <code><a href="localCheckpoint.html">localCheckpoint</a></code>, +<code><a href="merge.html">merge</a></code>, <code><a href="mutate.html">mutate</a></code>, +<code><a href="ncol.html">ncol</a></code>, <code><a href="nrow.html">nrow</a></code>, +<code><a href="persist.html">persist</a></code>, <code><a href="printSchema.html">printSchema</a></code>, +<code><a href="randomSplit.html">randomSplit</a></code>, <code><a href="rbind.html">rbind</a></code>, +<code><a href="registerTempTable-deprecated.html">registerTempTable</a></code>, <code><a href="rename.html">rename</a></code>, +<code><a href="repartition.html">repartition</a></code>, <code><a href="rollup.html">rollup</a></code>, +<code><a href="sample.html">sample</a></code>, <code><a href="saveAsTable.html">saveAsTable</a></code>, +<code><a href="schema.html">schema</a></code>, <code><a href="selectExpr.html">selectExpr</a></code>, +<code><a href="select.html">select</a></code>, <code><a href="showDF.html">showDF</a></code>, +<code><a href="show.html">show</a></code>, <code><a href="storageLevel.html">storageLevel</a></code>, +<code><a href="str.html">str</a></code>, <code><a href="subset.html">subset</a></code>, +<code><a href="summary.html">summary</a></code>, <code><a href="take.html">take</a></code>, +<code><a href="toJSON.html">toJSON</a></code>, <code><a href="unionByName.html">unionByName</a></code>, +<code><a href="union.html">union</a></code>, <code><a href="unpersist.html">unpersist</a></code>, +<code><a href="withColumn.html">withColumn</a></code>, <code><a href="withWatermark.html">withWatermark</a></code>, +<code><a href="with.html">with</a></code>, <code><a href="write.df.html">write.df</a></code>, +<code><a href="write.jdbc.html">write.jdbc</a></code>, <code><a href="write.json.html">write.json</a></code>, +<code><a href="write.orc.html">write.orc</a></code>, <code><a href="write.parquet.html">write.parquet</a></code>, +<code><a href="write.stream.html">write.stream</a></code>, <code><a href="write.text.html">write.text</a></code> +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D # Compute the average for all numeric columns grouped by department. +##D avg(groupBy(df, "department")) +##D +##D # Compute the max age and average salary, grouped by department and gender. +##D agg(groupBy(df, "department", "gender"), salary="avg", "age" -> "max") +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/hashCode.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/hashCode.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/hashCode.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,68 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: Compute the hashCode of an object</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for hashCode {SparkR}"><tr><td>hashCode {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>Compute the hashCode of an object</h2> + +<h3>Description</h3> + +<p>Java-style function to compute the hashCode for the given object. Returns +an integer value. +</p> + + +<h3>Usage</h3> + +<pre> +hashCode(key) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>key</code></td> +<td> +<p>the object to be hashed</p> +</td></tr> +</table> + + +<h3>Details</h3> + +<p>This only works for integer, numeric and character types right now. +</p> + + +<h3>Value</h3> + +<p>the hash code as an integer +</p> + + +<h3>Note</h3> + +<p>hashCode since 1.4.0 +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D hashCode(1L) # 1 +##D hashCode(1.0) # 1072693248 +##D hashCode("1") # 49 +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/head.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/head.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/head.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,115 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: Head</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for head {SparkR}"><tr><td>head {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>Head</h2> + +<h3>Description</h3> + +<p>Return the first <code>num</code> rows of a SparkDataFrame as a R data.frame. If <code>num</code> is not +specified, then head() returns the first 6 rows as with R data.frame. +</p> + + +<h3>Usage</h3> + +<pre> +## S4 method for signature 'SparkDataFrame' +head(x, num = 6L) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>x</code></td> +<td> +<p>a SparkDataFrame.</p> +</td></tr> +<tr valign="top"><td><code>num</code></td> +<td> +<p>the number of rows to return. Default is 6.</p> +</td></tr> +</table> + + +<h3>Value</h3> + +<p>A data.frame. +</p> + + +<h3>Note</h3> + +<p>head since 1.4.0 +</p> + + +<h3>See Also</h3> + +<p>Other SparkDataFrame functions: <code><a href="SparkDataFrame.html">SparkDataFrame-class</a></code>, +<code><a href="summarize.html">agg</a></code>, <code><a href="alias.html">alias</a></code>, +<code><a href="arrange.html">arrange</a></code>, <code><a href="as.data.frame.html">as.data.frame</a></code>, +<code><a href="attach.html">attach,SparkDataFrame-method</a></code>, +<code><a href="broadcast.html">broadcast</a></code>, <code><a href="cache.html">cache</a></code>, +<code><a href="checkpoint.html">checkpoint</a></code>, <code><a href="coalesce.html">coalesce</a></code>, +<code><a href="collect.html">collect</a></code>, <code><a href="columns.html">colnames</a></code>, +<code><a href="coltypes.html">coltypes</a></code>, +<code><a href="createOrReplaceTempView.html">createOrReplaceTempView</a></code>, +<code><a href="crossJoin.html">crossJoin</a></code>, <code><a href="cube.html">cube</a></code>, +<code><a href="dapplyCollect.html">dapplyCollect</a></code>, <code><a href="dapply.html">dapply</a></code>, +<code><a href="describe.html">describe</a></code>, <code><a href="dim.html">dim</a></code>, +<code><a href="distinct.html">distinct</a></code>, <code><a href="dropDuplicates.html">dropDuplicates</a></code>, +<code><a href="nafunctions.html">dropna</a></code>, <code><a href="drop.html">drop</a></code>, +<code><a href="dtypes.html">dtypes</a></code>, <code><a href="except.html">except</a></code>, +<code><a href="explain.html">explain</a></code>, <code><a href="filter.html">filter</a></code>, +<code><a href="first.html">first</a></code>, <code><a href="gapplyCollect.html">gapplyCollect</a></code>, +<code><a href="gapply.html">gapply</a></code>, <code><a href="getNumPartitions.html">getNumPartitions</a></code>, +<code><a href="groupBy.html">group_by</a></code>, <code><a href="hint.html">hint</a></code>, +<code><a href="histogram.html">histogram</a></code>, <code><a href="insertInto.html">insertInto</a></code>, +<code><a href="intersect.html">intersect</a></code>, <code><a href="isLocal.html">isLocal</a></code>, +<code><a href="isStreaming.html">isStreaming</a></code>, <code><a href="join.html">join</a></code>, +<code><a href="limit.html">limit</a></code>, <code><a href="localCheckpoint.html">localCheckpoint</a></code>, +<code><a href="merge.html">merge</a></code>, <code><a href="mutate.html">mutate</a></code>, +<code><a href="ncol.html">ncol</a></code>, <code><a href="nrow.html">nrow</a></code>, +<code><a href="persist.html">persist</a></code>, <code><a href="printSchema.html">printSchema</a></code>, +<code><a href="randomSplit.html">randomSplit</a></code>, <code><a href="rbind.html">rbind</a></code>, +<code><a href="registerTempTable-deprecated.html">registerTempTable</a></code>, <code><a href="rename.html">rename</a></code>, +<code><a href="repartition.html">repartition</a></code>, <code><a href="rollup.html">rollup</a></code>, +<code><a href="sample.html">sample</a></code>, <code><a href="saveAsTable.html">saveAsTable</a></code>, +<code><a href="schema.html">schema</a></code>, <code><a href="selectExpr.html">selectExpr</a></code>, +<code><a href="select.html">select</a></code>, <code><a href="showDF.html">showDF</a></code>, +<code><a href="show.html">show</a></code>, <code><a href="storageLevel.html">storageLevel</a></code>, +<code><a href="str.html">str</a></code>, <code><a href="subset.html">subset</a></code>, +<code><a href="summary.html">summary</a></code>, <code><a href="take.html">take</a></code>, +<code><a href="toJSON.html">toJSON</a></code>, <code><a href="unionByName.html">unionByName</a></code>, +<code><a href="union.html">union</a></code>, <code><a href="unpersist.html">unpersist</a></code>, +<code><a href="withColumn.html">withColumn</a></code>, <code><a href="withWatermark.html">withWatermark</a></code>, +<code><a href="with.html">with</a></code>, <code><a href="write.df.html">write.df</a></code>, +<code><a href="write.jdbc.html">write.jdbc</a></code>, <code><a href="write.json.html">write.json</a></code>, +<code><a href="write.orc.html">write.orc</a></code>, <code><a href="write.parquet.html">write.parquet</a></code>, +<code><a href="write.stream.html">write.stream</a></code>, <code><a href="write.text.html">write.text</a></code> +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D sparkR.session() +##D path <- "path/to/file.json" +##D df <- read.json(path) +##D head(df) +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> Added: dev/spark/v2.3.0-rc1-docs/_site/api/R/hint.html ============================================================================== --- dev/spark/v2.3.0-rc1-docs/_site/api/R/hint.html (added) +++ dev/spark/v2.3.0-rc1-docs/_site/api/R/hint.html Sat Jan 13 10:29:47 2018 @@ -0,0 +1,120 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>R: hint</title> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<link rel="stylesheet" type="text/css" href="R.css" /> + +<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/styles/github.min.css"> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/highlight.min.js"></script> +<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.3/languages/r.min.js"></script> +<script>hljs.initHighlightingOnLoad();</script> +</head><body> + +<table width="100%" summary="page for hint {SparkR}"><tr><td>hint {SparkR}</td><td style="text-align: right;">R Documentation</td></tr></table> + +<h2>hint</h2> + +<h3>Description</h3> + +<p>Specifies execution plan hint and return a new SparkDataFrame. +</p> + + +<h3>Usage</h3> + +<pre> +hint(x, name, ...) + +## S4 method for signature 'SparkDataFrame,character' +hint(x, name, ...) +</pre> + + +<h3>Arguments</h3> + +<table summary="R argblock"> +<tr valign="top"><td><code>x</code></td> +<td> +<p>a SparkDataFrame.</p> +</td></tr> +<tr valign="top"><td><code>name</code></td> +<td> +<p>a name of the hint.</p> +</td></tr> +<tr valign="top"><td><code>...</code></td> +<td> +<p>optional parameters for the hint.</p> +</td></tr> +</table> + + +<h3>Value</h3> + +<p>A SparkDataFrame. +</p> + + +<h3>Note</h3> + +<p>hint since 2.2.0 +</p> + + +<h3>See Also</h3> + +<p>Other SparkDataFrame functions: <code><a href="SparkDataFrame.html">SparkDataFrame-class</a></code>, +<code><a href="summarize.html">agg</a></code>, <code><a href="alias.html">alias</a></code>, +<code><a href="arrange.html">arrange</a></code>, <code><a href="as.data.frame.html">as.data.frame</a></code>, +<code><a href="attach.html">attach,SparkDataFrame-method</a></code>, +<code><a href="broadcast.html">broadcast</a></code>, <code><a href="cache.html">cache</a></code>, +<code><a href="checkpoint.html">checkpoint</a></code>, <code><a href="coalesce.html">coalesce</a></code>, +<code><a href="collect.html">collect</a></code>, <code><a href="columns.html">colnames</a></code>, +<code><a href="coltypes.html">coltypes</a></code>, +<code><a href="createOrReplaceTempView.html">createOrReplaceTempView</a></code>, +<code><a href="crossJoin.html">crossJoin</a></code>, <code><a href="cube.html">cube</a></code>, +<code><a href="dapplyCollect.html">dapplyCollect</a></code>, <code><a href="dapply.html">dapply</a></code>, +<code><a href="describe.html">describe</a></code>, <code><a href="dim.html">dim</a></code>, +<code><a href="distinct.html">distinct</a></code>, <code><a href="dropDuplicates.html">dropDuplicates</a></code>, +<code><a href="nafunctions.html">dropna</a></code>, <code><a href="drop.html">drop</a></code>, +<code><a href="dtypes.html">dtypes</a></code>, <code><a href="except.html">except</a></code>, +<code><a href="explain.html">explain</a></code>, <code><a href="filter.html">filter</a></code>, +<code><a href="first.html">first</a></code>, <code><a href="gapplyCollect.html">gapplyCollect</a></code>, +<code><a href="gapply.html">gapply</a></code>, <code><a href="getNumPartitions.html">getNumPartitions</a></code>, +<code><a href="groupBy.html">group_by</a></code>, <code><a href="head.html">head</a></code>, +<code><a href="histogram.html">histogram</a></code>, <code><a href="insertInto.html">insertInto</a></code>, +<code><a href="intersect.html">intersect</a></code>, <code><a href="isLocal.html">isLocal</a></code>, +<code><a href="isStreaming.html">isStreaming</a></code>, <code><a href="join.html">join</a></code>, +<code><a href="limit.html">limit</a></code>, <code><a href="localCheckpoint.html">localCheckpoint</a></code>, +<code><a href="merge.html">merge</a></code>, <code><a href="mutate.html">mutate</a></code>, +<code><a href="ncol.html">ncol</a></code>, <code><a href="nrow.html">nrow</a></code>, +<code><a href="persist.html">persist</a></code>, <code><a href="printSchema.html">printSchema</a></code>, +<code><a href="randomSplit.html">randomSplit</a></code>, <code><a href="rbind.html">rbind</a></code>, +<code><a href="registerTempTable-deprecated.html">registerTempTable</a></code>, <code><a href="rename.html">rename</a></code>, +<code><a href="repartition.html">repartition</a></code>, <code><a href="rollup.html">rollup</a></code>, +<code><a href="sample.html">sample</a></code>, <code><a href="saveAsTable.html">saveAsTable</a></code>, +<code><a href="schema.html">schema</a></code>, <code><a href="selectExpr.html">selectExpr</a></code>, +<code><a href="select.html">select</a></code>, <code><a href="showDF.html">showDF</a></code>, +<code><a href="show.html">show</a></code>, <code><a href="storageLevel.html">storageLevel</a></code>, +<code><a href="str.html">str</a></code>, <code><a href="subset.html">subset</a></code>, +<code><a href="summary.html">summary</a></code>, <code><a href="take.html">take</a></code>, +<code><a href="toJSON.html">toJSON</a></code>, <code><a href="unionByName.html">unionByName</a></code>, +<code><a href="union.html">union</a></code>, <code><a href="unpersist.html">unpersist</a></code>, +<code><a href="withColumn.html">withColumn</a></code>, <code><a href="withWatermark.html">withWatermark</a></code>, +<code><a href="with.html">with</a></code>, <code><a href="write.df.html">write.df</a></code>, +<code><a href="write.jdbc.html">write.jdbc</a></code>, <code><a href="write.json.html">write.json</a></code>, +<code><a href="write.orc.html">write.orc</a></code>, <code><a href="write.parquet.html">write.parquet</a></code>, +<code><a href="write.stream.html">write.stream</a></code>, <code><a href="write.text.html">write.text</a></code> +</p> + + +<h3>Examples</h3> + +<pre><code class="r">## Not run: +##D df <- createDataFrame(mtcars) +##D avg_mpg <- mean(groupBy(createDataFrame(mtcars), "cyl"), "mpg") +##D +##D head(join(df, hint(avg_mpg, "broadcast"), df$cyl == avg_mpg$cyl)) +## End(Not run) +</code></pre> + + +<hr /><div style="text-align: center;">[Package <em>SparkR</em> version 2.3.0 <a href="00Index.html">Index</a>]</div> +</body></html> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org