Repository: apex-malhar Updated Branches: refs/heads/master b217740c2 -> 278d799b0
APEXMALHAR-2167 json formatter documentation Project: http://git-wip-us.apache.org/repos/asf/apex-malhar/repo Commit: http://git-wip-us.apache.org/repos/asf/apex-malhar/commit/278d799b Tree: http://git-wip-us.apache.org/repos/asf/apex-malhar/tree/278d799b Diff: http://git-wip-us.apache.org/repos/asf/apex-malhar/diff/278d799b Branch: refs/heads/master Commit: 278d799b0f73a4519ea31f6f982319c9f99b3c2a Parents: b217740 Author: shubham <[email protected]> Authored: Wed Aug 3 12:02:58 2016 +0530 Committer: shubham <[email protected]> Committed: Tue Nov 8 13:15:04 2016 +0530 ---------------------------------------------------------------------- .../images/jsonFormatter/JsonFormatter.png | Bin 0 -> 29253 bytes docs/operators/jsonFormatter.md | 102 +++++++++++++++++++ mkdocs.yml | 1 + 3 files changed, 103 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/apex-malhar/blob/278d799b/docs/operators/images/jsonFormatter/JsonFormatter.png ---------------------------------------------------------------------- diff --git a/docs/operators/images/jsonFormatter/JsonFormatter.png b/docs/operators/images/jsonFormatter/JsonFormatter.png new file mode 100644 index 0000000..f457c68 Binary files /dev/null and b/docs/operators/images/jsonFormatter/JsonFormatter.png differ http://git-wip-us.apache.org/repos/asf/apex-malhar/blob/278d799b/docs/operators/jsonFormatter.md ---------------------------------------------------------------------- diff --git a/docs/operators/jsonFormatter.md b/docs/operators/jsonFormatter.md new file mode 100644 index 0000000..da01649 --- /dev/null +++ b/docs/operators/jsonFormatter.md @@ -0,0 +1,102 @@ +Json Formatter +============= + +## Operator Objective +Purpose of JsonFormatter is to consume Plain Old Java Object ("POJO") and write them as JSON. +Json Formatter is **idempotent**, **fault-tolerance** & **statically/dynamically partitionable**. + +## Class Diagram + +## Operator Information +1. Operator location:**_malhar-library** +2. Available since:**_3.2.0_** +3. Operator state:**_Evolving_** +4. Java Package:[com.datatorrent.lib.formatter.JsonFormatter](https://github.com/apache/apex-malhar/blob/master/library/src/main/java/com/datatorrent/lib/formatter/JsonFormatter.java) + +## Properties, Attributes and Ports + +### Platform Attributes that influences operator behavior +| **Attribute** | **Description** | **Type** | **Mandatory** | +| -------- | ----------- | ---- | ------------------ | +| *in.TUPLE_CLASS* | TUPLE_CLASS attribute on input port which tells operator the class of incoming POJO | Class or FQCN| Yes | + + +### Ports +| **Port** | **Description** | **Type** | **Mandatory** | +| -------- | ----------- | ---- | ------------------ | +| *in* | Tuples that needs to be formatted are recieved on this port | Object (POJO) | Yes +| *out* | Valid Tuples that are emitted as JSON | String | No | +| *err* | Invalid Tuples are emitted on this port | Object | No | + +## Partitioning +JSON Formatter is both statically and dynamically partitionable. +### Static Partitioning +This can be achieved in 2 ways + +1. Specifying the partitioner and number of partitions in the populateDAG() method + +```java +JsonFormatter jsonFormatter = dag.addOperator("jsonFormatter", JsonFormatter.class); +StatelessPartitioner<JsonFormatter> partitioner1 = new StatelessPartitioner<JsonFormatter>(2); +dag.setAttribute(jsonFormatter, Context.OperatorContext.PARTITIONER, partitioner1 ); +``` +2. Specifying the partitioner in properties file. +```xml + <property> + <name>dt.operator.{OperatorName}.attr.PARTITIONER</name> + <value>com.datatorrent.common.partitioner.StatelessPartitioner:2</value> + </property> +``` + where {OperatorName} is the name of the JsonFormatter operator. + Above lines will partition JsonFormatter statically 2 times. Above value can be changed accordingly to change the number of static partitions. + + +### Dynamic Paritioning + +JsonFormatter can be dynamically partitioned using an out-of-the-box partitioner: + +#### Throughput based +Following code can be added to populateDAG method of application to dynamically partition JsonFormatter: +```java +JsonFormatter jsonFormatter = dag.addOperator("jsonFormatter", JsonFormatter.class); +StatelessThroughputBasedPartitioner<JsonFormatter> partitioner = new StatelessThroughputBasedPartitioner<>(); +partitioner.setCooldownMillis(conf.getLong(COOL_DOWN_MILLIS, 10000)); +partitioner.setMaximumEvents(conf.getLong(MAX_THROUGHPUT, 30000)); +partitioner.setMinimumEvents(conf.getLong(MIN_THROUGHPUT, 10000)); +dag.setAttribute(JsonFormatter, OperatorContext.STATS_LISTENERS, Arrays.asList(new StatsListener[]{partitioner})); +dag.setAttribute(JsonFormatter, OperatorContext.PARTITIONER, partitioner); +``` + +Above code will dynamically partition JsonFormatter when the throughput changes. +If the overall throughput of JsonFormatter goes beyond 30000 or less than 10000, the platform will repartition JsonFormatter +to balance throughput of a single partition to be between 10000 and 30000. +CooldownMillis of 10000 will be used as the threshold time for which the throughput change is observed. + +## Example +Example for Json Formatter can be found at: [https://github.com/DataTorrent/examples/tree/master/tutorials/parser](https://github.com/DataTorrent/examples/tree/master/tutorials/parser) + +## Advance Features +JsonFormatter is based on [jackson-databind](https://github.com/FasterXML/jackson-databind) and so users can make use of [annotations](https://github.com/FasterXML/jackson-annotations) in POJO class. Here are few annotations that are relavant while using JsonFormatter +1. **@JsonProperty** : Sometimes POJOs contain properties that has different name from incoming POJOs.You can specify names as: +```java +public class Ad{ + @JsonProperty("desc") + public String description; + public List<String> sizes; +} +``` +2. **@JsonIgnore** : Sometimes POJOs contain properties that you do not want to write out, so you can do: +```java +public class Value { + public int value; + @JsonIgnore + public int internalValue; +} +``` +3. **@JsonFormat** : Sometimes Date fields need to be printed in custom format, so you can do: +```java +public class Ad{ + @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "EEE, d MMM yyyy HH:mm:ss") + public Date startDate; +} +``` http://git-wip-us.apache.org/repos/asf/apex-malhar/blob/278d799b/mkdocs.yml ---------------------------------------------------------------------- diff --git a/mkdocs.yml b/mkdocs.yml index e18aaea..9226703 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -15,4 +15,5 @@ pages: - Deduper: operators/deduper.md - Windowed Operator: operators/windowedOperator.md - Json Parser: operators/jsonParser.md + - Json Formatter: operators/jsonFormatter.md
