jon-wei commented on a change in pull request #6119: Add docs for virtual
columns and transform specs
URL: https://github.com/apache/incubator-druid/pull/6119#discussion_r208760623
##########
File path: docs/content/ingestion/transform-spec.md
##########
@@ -0,0 +1,64 @@
+---
+layout: doc_page
+---
+
+# Transform Specs
+
+Transform specs allow Druid to filter and transform input data during
ingestion.
+
+## Syntax
+
+The syntax for the transformSpec is shown below:
+
+```
+"transformSpec": {
+ "transforms: <List of transforms>,
+ "filter": <filter>
+}
+```
+
+|property|description|required?|
+|--------|-----------|---------|
+|transforms|A list of [transforms](#transforms) to be applied to input rows.
|no|
+|filter|A [filter](../querying/filters.html) that will be applied to input
rows; only rows that pass the filter will be ingested.|no|
+
+## Transforms
+
+The `transforms` list allows the user to specify a set of column
transformations to be performed on input data.
+
+Transforms allow adding new fields to input rows. Each transform has a "name"
(the name of the new field) which can be referred to by DimensionSpecs,
AggregatorFactories, etc.
+
+A transform behaves as a "row function", taking an entire row as input and
outputting a column value.
+
+If a transform has the same name as a field in an input row, then it will
shadow the original field. Transforms that shadow fields may still refer to the
fields they shadow. This can be used to transform a field "in-place".
+
+Transforms do have some limitations. They can only refer to fields present in
the actual input rows; in particular, they cannot refer to other transforms.
And they cannot remove fields, only add them. However, they can shadow a field
with another field containing all nulls, which will act similarly to removing
the field.
+
+Note that the transforms are applied before the filter.
+
+### Expression Transform
+
+Druid currently supports one kind of transform, the expression transform.
+
+An expression transform has the following syntax:
+
+```
+{
+ "type": "expression",
+ "name": <output field name>,
+ "expression": <expr>
+}
+```
+
+|property|description|required?|
+|--------|-----------|---------|
+|name|The output field name of the expression transform.|yes|
+|expression|An [expression](../misc/math-expr.html) that will be applied to
input rows to produce a value for the transform's output field.|no|
+
Review comment:
Added a transform example and a filter example, I plan on doing a tutorial
eventually for this but haven't had time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
