[
https://issues.apache.org/jira/browse/NIFI-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322550#comment-15322550
]
Bryan Bende edited comment on NIFI-1935 at 6/9/16 2:06 PM:
-----------------------------------------------------------
[~Daniel Cave] [~ahalldin]
Was looking this over, and admittedly haven't done a full review, but just
wanted to mention something... the description says that part of the reason for
creating this processor was because ConverJsonToAvro requires a predefined
schema, but I'm not sure that is the case...
The schema property in ConvertJsonToAvro supports expression language:
{code}
static final PropertyDescriptor SCHEMA
= new PropertyDescriptor.Builder()
.name("Record schema")
.description("Outgoing Avro schema for each record created from a
JSON object")
.addValidator(SCHEMA_VALIDATOR)
.expressionLanguageSupported(true)
.required(true)
.build();
{code}
And InferAvroSchema has a property to select where to store the schema -
attributes vs. content. So if you choose attributes, it creates an attribute on
the FlowFile called "inferred.avro.schema" which contains the schema as the
value, so now you have a single flow file with the JSON as the content and the
schema in the attributes.
You could then set the SCHEMA property in ConvertJsonToAvro to
{code}
${inferred.avro.schema}
{code}
to reference the schema, so each incoming flow file could have a different
schema.
Given the above, do you think ConvertDynamicJsonToAvro still provides
additional benefits over ConvertJsonToAvro?
was (Author: bende):
[~Daniel Cave] [~ahalldin]
Was looking this over, and admittedly haven't done a full review, but just
wanted to mention something... the description says that part of the reason for
creating this processor was because ConverJsonToAvro requires a predefined
schema, but I'm not sure that is the case...
The schema property in ConvertJsonToAvro supports expression language:
{code}
static final PropertyDescriptor SCHEMA
= new PropertyDescriptor.Builder()
.name("Record schema")
.description("Outgoing Avro schema for each record created from a
JSON object")
.addValidator(SCHEMA_VALIDATOR)
.expressionLanguageSupported(true)
.required(true)
.build();
{code}
And InferAvroSchema has a property to select where to store the schema -
attributes vs. content. So if you choose attributes, it creates an attribute on
the FlowFile called "inferred.avro.schema" which contains the schema as the
value, so now you have a single flow file with the JSON as the content and the
schema in the attributes. You could then set the SCHEMA property in
ConvertJsonToAvro to ${inferred.avro.schema} to reference the schema, so each
incoming flow file could have a different schema.
Given the above, do you think ConvertDynamicJsonToAvro still provides
additional benefits over ConvertJsonToAvro?
> Added ConvertDynamicJsonToAvro processor
> ----------------------------------------
>
> Key: NIFI-1935
> URL: https://issues.apache.org/jira/browse/NIFI-1935
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
> Affects Versions: 1.0.0, 0.7.0
> Reporter: Daniel Cave
> Assignee: Alex Halldin
> Priority: Minor
> Fix For: 1.0.0, 0.7.0
>
> Attachments:
> 0001-NIFI-1935-Added-ConvertDynamicJSONToAvro.java.-Added.patch
>
>
> ConvertJsonToAvro required a predefined Avro schema to convert JSON and
> required the presence of all field on the incoming JSON.
> ConvertDynamicJsonToAvro functions similarly, however it now accepts the JSON
> and schema as incoming flowfiles and creates the Avro dynamically.
> This processor requires the InferAvroSchema processor in its upstream flow so
> that it can use the original and schema flowfiles as input. These two
> flowfiles will have the unique attribute inferredAvroId set on them by
> InferAvroSchema so that they can be properly matched in
> ConvertDynamicJsonToAvro.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
