This is an automated email from the ASF dual-hosted git repository. dweiss pushed a commit to branch jira/solr-13105-toMerge in repository https://gitbox.apache.org/repos/asf/solr.git
commit edc16eb58bc9ac1c460d3233e7527c02b152e329 Author: Cassandra Targett <[email protected]> AuthorDate: Thu Jan 14 11:37:08 2021 -0600 Fix page order in side bar; probably final last big round of copy edit changes --- solr/solr-ref-guide/src/loading.adoc | 16 ++++----- .../src/major-changes-in-solr-8.adoc | 2 +- solr/solr-ref-guide/src/math-expressions.adoc | 6 ++-- solr/solr-ref-guide/src/math-start.adoc | 42 +++++++++++----------- solr/solr-ref-guide/src/numerical-analysis.adoc | 2 +- .../src/stream-decorator-reference.adoc | 8 ++--- .../src/stream-evaluator-reference.adoc | 14 ++++---- .../src/stream-source-reference.adoc | 24 +++++++------ solr/solr-ref-guide/src/streaming-expressions.adoc | 28 ++++++++------- solr/solr-ref-guide/src/time-series.adoc | 5 ++- solr/solr-ref-guide/src/transform.adoc | 4 +-- 11 files changed, 76 insertions(+), 75 deletions(-) diff --git a/solr/solr-ref-guide/src/loading.adoc b/solr/solr-ref-guide/src/loading.adoc index 3017eba..5c2db31 100644 --- a/solr/solr-ref-guide/src/loading.adoc +++ b/solr/solr-ref-guide/src/loading.adoc @@ -16,15 +16,13 @@ // specific language governing permissions and limitations // under the License. -Streaming Expressions has support for reading, parsing, transforming, visualizing -and loading CSV and TSV formatted data. These functions are designed to cut down the -time spent on data preparation and allow users to begin data exploration before the data is -loaded into Solr. +Streaming expressions has support for reading, parsing, transforming, visualizing and loading CSV and TSV formatted data. +These functions are designed to cut down the time spent on data preparation and allow users to begin data exploration before the data is loaded into Solr. == Reading Files The `cat` function can be used to read files under the *userfiles* directory in -$SOLR_HOME. The `cat` function takes two parameters. +`$SOLR_HOME`. The `cat` function takes two parameters. The first parameter is a comma-delimited list of paths. If the path list contains directories, `cat` will crawl all the files in the directory and sub-directories. @@ -187,9 +185,9 @@ image::images/math-expressions/update.png[] == Transforming Data -Streaming Expressions and Math Expression provide a powerful set of functions -for transforming data. The section below shows some useful transformations that -can be applied while analyzing, visualizing, and loading CSV and TSV files. +Streaming expressions and math expressions provide a powerful set of functions +for transforming data. +The section below shows some useful transformations that can be applied while analyzing, visualizing, and loading CSV and TSV files. === Unique IDs @@ -474,7 +472,7 @@ more compute resources to be applied to the NLP indexing then is available on th * Using the `cartesianProduct` function the analyzed tokens can be indexed as individual documents which allows analyzed tokens to be searched and analyzed with Solr's aggregation and graph expressions. * Also using `cartesianProduct` the analyzed tokens can be aggregated, analyzed and visualized using -Streaming Expressions directly before indexing occurs. +streaming expressions directly before indexing occurs. Below is an example of the `analyze` function being applied to the *Resolution.Description* diff --git a/solr/solr-ref-guide/src/major-changes-in-solr-8.adoc b/solr/solr-ref-guide/src/major-changes-in-solr-8.adoc index 9bb805d..2ed72ae 100644 --- a/solr/solr-ref-guide/src/major-changes-in-solr-8.adoc +++ b/solr/solr-ref-guide/src/major-changes-in-solr-8.adoc @@ -489,7 +489,7 @@ See the section <<metrics-reporting.adoc#shard-and-cluster-reporters,Shard and C *Streaming Expressions* -* All Stream Evaluators in `solrj.io.eval` have been refactored to have a simpler and more robust structure. This simplifies and condenses the code required to implement a new Evaluator and makes it much easier for evaluators to handle differing data types (primitives, objects, arrays, lists, and so forth). +* All stream evaluators in `solrj.io.eval` have been refactored to have a simpler and more robust structure. This simplifies and condenses the code required to implement a new Evaluator and makes it much easier for evaluators to handle differing data types (primitives, objects, arrays, lists, and so forth). *ReplicationHandler* diff --git a/solr/solr-ref-guide/src/math-expressions.adoc b/solr/solr-ref-guide/src/math-expressions.adoc index af79bc7..987c627 100644 --- a/solr/solr-ref-guide/src/math-expressions.adoc +++ b/solr/solr-ref-guide/src/math-expressions.adoc @@ -1,5 +1,5 @@ = Streaming Expressions and Math Expressions -:page-children: visualization, math-start, loading, search-sample, transform, scalar-math, vector-math, variables, matrix-math, term-vectors, probability-distributions, statistics, simulations, time-series, regression, numerical-analysis, curve-fitting, dsp, machine-learning, computational-geometry, logs +:page-children: visualization, math-start, loading, search-sample, transform, scalar-math, vector-math, variables, matrix-math, term-vectors, probability-distributions, statistics, regression, curve-fitting, time-series, numerical-analysis, dsp, simulations, machine-learning, computational-geometry, logs :page-show-toc: false // Licensed to the Apache Software Foundation (ASF) under one @@ -25,9 +25,9 @@ image::images/math-expressions/searchiris.png[] == Table of Contents -*<<visualization.adoc#visualization,Visualizations>>*: Gallery of Streaming Expressions and Math Expressions visualizations. +*<<visualization.adoc#visualization,Visualizations>>*: Gallery of streaming expression and math expression visualizations. -*<<math-start.adoc#math-start,Getting Started>>*: Getting started with Streaming Expressions, Math Expressions and visualization. +*<<math-start.adoc#math-start,Getting Started>>*: Getting started with streaming expressions, math expressions, and visualization. *<<loading.adoc#data-loading,Data Loading>>*: Visualizing, transforming and loading CSV files. diff --git a/solr/solr-ref-guide/src/math-start.adoc b/solr/solr-ref-guide/src/math-start.adoc index ca79113..63879ba 100644 --- a/solr/solr-ref-guide/src/math-start.adoc +++ b/solr/solr-ref-guide/src/math-start.adoc @@ -18,40 +18,38 @@ == Language -*Streaming Expressions* and *Math Expressions* are function languages that run +*Streaming expressions* and *math expressions* are function languages that run inside SolrCloud. The languages consist of functions that are designed to be *composed* to form programming logic. -*Streaming Expressions* are functions that return streams of tuples. Streaming Expression functions can be -composed to form a transformation pipeline. The pipeline starts with a *Stream Source*, such as *search*, -which initiates a stream of tuples. One or more *Stream Decorators*, such as *select*, wraps the Stream Source and transforms -the stream of Tuples. +*Streaming expressions* are functions that return streams of tuples. Streaming expression functions can be composed to form a transformation pipeline. +The pipeline starts with a *stream source*, such as `search`, which initiates a stream of tuples. +One or more *stream decorators*, such as `select`, wraps the stream source and transforms the stream of tuples. -*Math Expressions* are functions that operate over and return primitives and in-memory -arrays and matrices. The core use case for Math Expressions is performing mathematical operations and +*Math expressions* are functions that operate over and return primitives and in-memory +arrays and matrices. The core use case for math expressions is performing mathematical operations and visualization. -Streaming Expressions and Math Expressions can be combined to *search, +Streaming expressions and math expressions can be combined to *search, sample, aggregate, transform, analyze* and *visualize* data in SolrCloud collections. == Execution -Solr's `/stream` request handler executes Streaming Expressions and Math Expressions. +Solr's `/stream` request handler executes streaming expressions and math expressions. This handler compiles the expression, runs the expression logic and returns a JSON result. === Admin UI Stream Panel -The easiest way to run Streaming Expressions and Math expressions is through -the *stream* panel on the Solr Admin -UI. +The easiest way to run streaming expressions and math expressions is through +the *stream* panel on the Solr Admin UI. -A sample *search* Streaming Expression is shown in the screenshot below: +A sample `search` streaming expression is shown in the screenshot below: image::images/math-expressions/search.png[] -A sample *add* Math Expression is shown in the screenshot below: +A sample `add` math expression is shown in the screenshot below: image::images/math-expressions/add.png[] @@ -103,7 +101,7 @@ Zeppelin-Solr interpreter. === Zeppelin-Solr Interpreter -A Zeppelin interpreter for Solr allows Streaming Expressions and Math Expressions to be executed and results visualized in Apache Zeppelin. +An Apache Zeppelin interpreter for Solr allows streaming expressions and math expressions to be executed and results visualized in Zeppelin. The instructions for installing and configuring Zeppelin-Solr can be found on the Github repository for the project: https://github.com/lucidworks/zeppelin-solr @@ -112,19 +110,19 @@ The screenshot below shows the panel for configuring Zeppelin-Solr. image::images/math-expressions/zepconf.png[] -Configure the `solr.baseUrl` and `solr.collection` to point to the location where the Streaming -Expressions and Math Expressions will be sent for execution. The `solr.collection` is +Configure the `solr.baseUrl` and `solr.collection` to point to the location where the streaming +expressions and math expressions will be sent for execution. The `solr.collection` is just the execution collection and does not need to hold data, although it can hold data. -Streaming Expressions can choose to query any of the collections that are attached +streaming expressions can choose to query any of the collections that are attached to the same SolrCloud as the execution collection. === zplot -Streaming Expression result sets can be visualized automatically by Zeppelin-Solr. +Streaming expression result sets can be visualized automatically by Zeppelin-Solr. -Math Expression results need to be formatted for visualization using the `zplot` function. +Math expression results need to be formatted for visualization using the `zplot` function. This function has support for plotting *vectors*, *matrices*, *probability distributions* and *2D clustering results*. -There are many examples in the guide which show how to visualize both Streaming Expressions -and Math Expressions. +There are many examples in the guide which show how to visualize both streaming expressions +and math expressions. diff --git a/solr/solr-ref-guide/src/numerical-analysis.adoc b/solr/solr-ref-guide/src/numerical-analysis.adoc index 9b1195e..dc3af58 100644 --- a/solr/solr-ref-guide/src/numerical-analysis.adoc +++ b/solr/solr-ref-guide/src/numerical-analysis.adoc @@ -16,7 +16,7 @@ // specific language governing permissions and limitations // under the License. -This section explores the interrelated Math Expressions for Interpolation and Numerical Calculus. +This section explores the interrelated math expressions for interpolation and numerical calculus. == Interpolation diff --git a/solr/solr-ref-guide/src/stream-decorator-reference.adoc b/solr/solr-ref-guide/src/stream-decorator-reference.adoc index ff7d95b..9c5063b 100644 --- a/solr/solr-ref-guide/src/stream-decorator-reference.adoc +++ b/solr/solr-ref-guide/src/stream-decorator-reference.adoc @@ -637,8 +637,8 @@ Users who anticipate concurrent updates, and wish to "skip" any failed deletes, The `eval` function allows for use cases where new streaming expressions are generated on the fly and then evaluated. The `eval` function wraps a streaming expression and reads a single tuple from the underlying stream. -The `eval` function then retrieves a string Streaming Expressions from the `expr_s` field of the tuple. -The `eval` function then compiles the string Streaming Expression and emits the tuples. +The `eval` function then retrieves a string streaming expression from the `expr_s` field of the tuple. +The `eval` function then compiles the string streaming expression and emits the tuples. === eval Parameters @@ -652,7 +652,7 @@ eval(expr) ---- In the example above the `eval` expression reads the first tuple from the underlying expression. It then compiles and -executes the string Streaming Expression in the expr_s field. +executes the string streaming expression in the `expr_s` field. == executor @@ -665,7 +665,7 @@ This model allows for asynchronous execution of jobs where the output is stored === executor Parameters * `threads`: (Optional) The number of threads in the executors thread pool for executing expressions. -* `StreamExpression`: (Mandatory) The stream source which contains the Streaming Expressions to execute. +* `StreamExpression`: (Mandatory) The stream source which contains the streaming expressions to execute. === executor Syntax diff --git a/solr/solr-ref-guide/src/stream-evaluator-reference.adoc b/solr/solr-ref-guide/src/stream-evaluator-reference.adoc index b1ef9ca..44efd5d 100644 --- a/solr/solr-ref-guide/src/stream-evaluator-reference.adoc +++ b/solr/solr-ref-guide/src/stream-evaluator-reference.adoc @@ -293,7 +293,7 @@ if(gt(fieldA,fieldB),ceil(fieldA),ceil(fieldB)) // if fieldA > fieldB then retur == col -The `col` function returns a numeric array from a list of Tuples. The `col` +The `col` function returns a numeric array from a list of tuples. The `col` function is used to create numeric arrays from stream sources. === col Parameters @@ -1020,11 +1020,11 @@ string array: The labels for each row in the matrix == getValue -The `getValue` function returns the value of a single Tuple entry by key. +The `getValue` function returns the value of a single tuple entry by key. === getValue Parameters -* `tuple`: The Tuple to return the entry from. +* `tuple`: The tuple to return the entry from. * `key`: The key of the entry to return the value for. === getValue Syntax @@ -1033,7 +1033,7 @@ getValue(tuple, key) === getValue Returns -object: Returns an object of the same type as the Tuple entry. +object: Returns an object of the same type as the tuple entry. == grandSum @@ -1589,7 +1589,7 @@ not(eq(fieldA,fieldB)) // true if fieldA != fieldB The `olsRegress` function performs https://en.wikipedia.org/wiki/Ordinary_least_squares[ordinary least squares], multivariate, linear regression. -The `olsRegress` function returns a single Tuple containing the regression model with estimated regression parameters, RSquared and regression diagnostics. +The `olsRegress` function returns a single tuple containing the regression model with estimated regression parameters, RSquared and regression diagnostics. The output of `olsRegress` can be used with the <<predict>> function to predict values based on the regression model. @@ -2082,11 +2082,11 @@ matrix: The matrix with the labels set. == setValue -The `setValue` function sets a new value for a Tuple entry. +The `setValue` function sets a new value for a tuple entry. === setValue Parameters -* `tuple`: The Tuple to return the entry from. +* `tuple`: The tuple to return the entry from. * `key`: The key of the entry to set. * `value`: The value to set. diff --git a/solr/solr-ref-guide/src/stream-source-reference.adoc b/solr/solr-ref-guide/src/stream-source-reference.adoc index d31cc3c..62bd4f6 100644 --- a/solr/solr-ref-guide/src/stream-source-reference.adoc +++ b/solr/solr-ref-guide/src/stream-source-reference.adoc @@ -110,11 +110,11 @@ jdbc( == drill -The `drill` function is designed to support efficient high cardinality aggregation. The `drill` -function sends a request to the `export` handler in a specific collection which includes a Streaming -Expression that the `export` handler applies to the sorted result set. The `export` handler then emits the aggregated tuples. -The `drill` function reads and emits the aggregated tuples fromn each shard maintaining the sort order, -but does not merge the aggregations. Streaming Expression functions can be wrapped around the `drill` function to +The `drill` function is designed to support efficient high cardinality aggregation. +The `drill` function sends a request to the `export` handler in a specific collection which includes a streaming expression that the `export` handler applies to the sorted result set. +The `export` handler then emits the aggregated tuples. +The `drill` function reads and emits the aggregated tuples from each shard maintaining the sort order, but does not merge the aggregations. +Streaming expression functions can be wrapped around the `drill` function to merge the aggregates. === drill Parameters @@ -154,8 +154,8 @@ rollup(drill(articles, == echo -The `echo` function returns a single Tuple echoing its text parameter. `Echo` is the simplest stream source designed to provide text -to a text analyzing stream decorator. +The `echo` function returns a single tuple echoing its text parameter. +`Echo` is the simplest stream source designed to provide text to a text analyzing stream decorator. === echo Syntax @@ -605,10 +605,12 @@ topic(checkpointCollection, == tuple -The `tuple` function emits a single Tuple with name/value pairs. The values can be set to variables assigned in a `let` expression, literals, Stream Evaluators or -Stream Expressions. In the case of Stream Evaluators the tuple will output the return value from the evaluator. -This could be a numeric, list or map. If a value is set to a Stream Expression, the `tuple` function will flatten -the tuple stream from the Stream Expression into a list of Tuples. +The `tuple` function emits a single tuple with name/value pairs. +The values can be set to variables assigned in a `let` expression, literals, stream evaluators or stream expressions. +In the case of stream evaluators the tuple will output the return value from the evaluator. +This could be a numeric, list, or map. +If a value is set to a stream expression, the `tuple` function will flatten +the tuple stream from the stream expression into a list of tuples. === tuple Parameters diff --git a/solr/solr-ref-guide/src/streaming-expressions.adoc b/solr/solr-ref-guide/src/streaming-expressions.adoc index af448b2..1ce3638 100644 --- a/solr/solr-ref-guide/src/streaming-expressions.adoc +++ b/solr/solr-ref-guide/src/streaming-expressions.adoc @@ -17,8 +17,8 @@ // specific language governing permissions and limitations // under the License. -Streaming Expressions exposes the capabilities of SolrCloud as composable functions. These functions provide a system for -searching, transforming, analyzing and visualizing data stored in SolrCloud collections. +Streaming expressions exposes the capabilities of SolrCloud as composable functions. +These functions provide a system for searching, transforming, analyzing, and visualizing data stored in SolrCloud collections. At a high level there a four main capabilities that will be explored in the documentation: @@ -33,9 +33,11 @@ At a high level there a four main capabilities that will be explored in the docu == Stream Language Basics -Streaming Expressions are comprised of streaming functions which work with a Solr collection. They emit a stream of tuples (key/value Maps). +Streaming expressions are comprised of streaming functions which work with a Solr collection. +They emit a stream of tuples (key/value Maps). -Some of the provided streaming functions are designed to work with entire result sets rather than the top N results like normal search. This is supported by the <<exporting-result-sets.adoc#exporting-result-sets,/export handler>>. +Some of the provided streaming functions are designed to work with entire result sets rather than the top N results like normal search. +This is supported by the <<exporting-result-sets.adoc#exporting-result-sets,/export handler>>. Some streaming functions act as stream sources to originate the stream flow. Other streaming functions act as stream decorators to wrap other stream functions and perform operations on the stream of tuples. Many streams functions can be parallelized across a worker collection. This can be particularly powerful for relational algebra functions. @@ -99,21 +101,23 @@ A full reference to all available decorator expressions is available in <<stream === Math Expressions Math expressions are a vector and matrix math library that can be combined with streaming expressions to perform analysis and build mathematical models -of the result sets. From a language standpoint Math Expressions are a sub-langauge of streaming expressions that don't return streams of tuples. Instead -they operate on and return numbers, vectors, matrices and mathematical models. The documentation will show how to combine Streaming Expressions and Math -Expressions. +of the result sets. +From a language standpoint math expressions are a sub-language of streaming expressions that don't return streams of tuples. +Instead they operate on and return numbers, vectors, matrices and mathematical models. +The documentation will show how to combine streaming expressions and math +expressions. -The Math Expressions user guide is available in <<>> +The math expressions user guide is available in <<>> -From a language standpoint Math Expressions are referred to as Stream Evaluators. +From a language standpoint math expressions are referred to as *stream evaluators*. A full reference to all available evaluator expressions is available in <<stream-evaluator-reference.adoc#stream-evaluator-reference,Stream Evaluator Reference>>. === Visualization -Visualization of both Streaming Expressions and Math Expressions is done using Apache Zeppelin and the Zeppelin-Solr Interpreter. +Visualization of both streaming expressions and math expressions is done using Apache Zeppelin and the Zeppelin-Solr Interpreter. -Visualizing Streaming expressions and setting up of Apache Zeppelin is documented in<<>> +Visualizing Streaming expressions and setting up of Apache Zeppelin is documented in <<math-start.adoc#zeppelin-solr-interpreter,Zeppelin-Solr Interpreter>>. -The Math Expressions user guide has in depth coverage of visualization techniques. +The <<math-expressions.adoc,Math Expressions User Guide>> has in depth coverage of visualization techniques. diff --git a/solr/solr-ref-guide/src/time-series.adoc b/solr/solr-ref-guide/src/time-series.adoc index 60229be..abd920c 100644 --- a/solr/solr-ref-guide/src/time-series.adoc +++ b/solr/solr-ref-guide/src/time-series.adoc @@ -277,9 +277,8 @@ image::images/math-expressions/outliers.png[] == Modeling -Math Expressions has a number of functions that can be used to -model a time series. These functions include linear regression, -polynomial and harmonic curve fitting, loess regression, and KNN regression. +Math expressions support in Solr includes a number of functions that can be used to model a time series. +These functions include linear regression, polynomial and harmonic curve fitting, loess regression, and KNN regression. Each of these functions can model a time series and be used for interpolation (predicting values within the dataset) and several diff --git a/solr/solr-ref-guide/src/transform.adoc b/solr/solr-ref-guide/src/transform.adoc index fe2bc30..61e5e06 100644 --- a/solr/solr-ref-guide/src/transform.adoc +++ b/solr/solr-ref-guide/src/transform.adoc @@ -17,8 +17,8 @@ // under the License. -Streaming Expressions provides a powerful set of functions for transforming result sets. This section -of the user guide provides an overview of useful transformations applied to result sets. +Streaming expressions provides a powerful set of functions for transforming result sets. +This section of the user guide provides an overview of useful transformations applied to result sets. == Selecting and Adding Fields
