One thing about its limitations had to do with timing, the record-aware stuff 
happened after QDT. Would be great to have QDT use a record writer, then 
depending on the writer you could choose your schema output strategy as Koji 
outlined.

I'm not sure if there is a JIRA for this or not (or any of the other earlier 
processors like ExecuteSQL), but we can certainly go after record-aware 
versions of them to get all the goodness therein.

Regards,
Matt 

Sent from my iPhone

> On Sep 11, 2017, at 9:05 PM, Koji Kawamura <ijokaruma...@gmail.com> wrote:
> 
> Hi Uwe,
> 
> I had a similar expectation when I was using QueryDatabaseTable or any
> other processor creating Avro FlowFile which has its schema embedded,
> combining new record reader/writer controllers.
> 
> Now, NiFi has "Inherit Record Schema" option as "Schema Access
> Strategy" of RecordWriter, already merged in master branch.
> https://issues.apache.org/jira/browse/NIFI-3921
> 
> I was able to reuse the Avro schema at subsequent flow using "Inherit
> Record Schema", it's really useful. You can construct a flow like
> below:
> 
> - QueryDatabaseTable
>  - outputs FlowFile with Avro schema embedded
> - ConvertRecord
>  - AvroReader:
>    - "Schema Access Strategy" = "Use Embedded Avro Schema"
>  - CSVRecordSetWriter:
>    - "Schema Access Strategy" = "Inherit Record Schema"
>    - "Schema Write Strategy" = "Set 'avro.schema' Attribute"
> 
> This way, you don't have to have the schema in registry, and result
> CSV FlowFile has 'avro.schema' attribute inheriting the one created by
> QueryDatabaseTable.
> 
> Hope this helps.
> 
> Thanks,
> Koji
> 
>> On Tue, Sep 12, 2017 at 5:02 AM, Uwe Geercken <uwe.geerc...@web.de> wrote:
>> Hello,
>> 
>> I was wondering why if the QueryDatabaseTable processor creates internally
>> an Avro schema, why is this schema not available as an attribute or saved to
>> the registry?
>> 
>> If it would, then one could reuse the schema. E.g. if I use the
>> ConvertRecord processor and I specify an AvroReader as RecordReader, then
>> this Reader will take the schema from the flowfile the QueryDatabaseTable
>> processor creates. But the RecordWriter in the ConvertRecord - in my example
>> a CSVRecordSetWriter requires the schema as an attribute or as a reference
>> to the schema registry.
>> 
>> I can see there is an ExtractAvroSchema processor but I don'see there is a
>> way of combining the metadata into e.g. the ConvertRecord processor.
>> 
>> Any help or ideas?
>> 
>> Rgds,
>> 
>> Uwe

Reply via email to