Thank you Koji!
that is good news. But I have 3 questions:
1. You quote Bryan Bende: "When a reader produces a record it attaches the schema it used to the record...": What happens here exactly? Is the schema attached to the flowfile? Is it an attribute?
2. I can not see an exact definition of what "inherit" means. It may be linked to my question above though. I am a bit puzzled of the use of "embedded" versus "inherit". Does it not mean "embedded" in both cases? If it really means inherit, from where does it inherit? or can I choose it?
3. What if I do really want to save the schema of e.g. a database table or file to the registry. I don't know maybe as a reference or for debugging. How would I do that (I mean: not manually)?
From the first look I found Nifi a kick-ass tool. It continues to evolve very fast and I use it at work for smaller things. Now I want to start to use it for more challenging things such as feeding kafka and maybe also hadoop. So I am experimenting a lot and want to find the best possible setup.
Greetings ans thanks again.
Uwe
Gesendet: Dienstag, 12. September 2017 um 03:05 Uhr
Von: "Koji Kawamura" <ijokaruma...@gmail.com>
An: users@nifi.apache.org
Betreff: Re: QueryDatabaseTable - Schema
Von: "Koji Kawamura" <ijokaruma...@gmail.com>
An: users@nifi.apache.org
Betreff: Re: QueryDatabaseTable - Schema
Hi Uwe,
I had a similar expectation when I was using QueryDatabaseTable or any
other processor creating Avro FlowFile which has its schema embedded,
combining new record reader/writer controllers.
Now, NiFi has "Inherit Record Schema" option as "Schema Access
Strategy" of RecordWriter, already merged in master branch.
https://issues.apache.org/jira/browse/NIFI-3921
I was able to reuse the Avro schema at subsequent flow using "Inherit
Record Schema", it's really useful. You can construct a flow like
below:
- QueryDatabaseTable
- outputs FlowFile with Avro schema embedded
- ConvertRecord
- AvroReader:
- "Schema Access Strategy" = "Use Embedded Avro Schema"
- CSVRecordSetWriter:
- "Schema Access Strategy" = "Inherit Record Schema"
- "Schema Write Strategy" = "Set 'avro.schema' Attribute"
This way, you don't have to have the schema in registry, and result
CSV FlowFile has 'avro.schema' attribute inheriting the one created by
QueryDatabaseTable.
Hope this helps.
Thanks,
Koji
On Tue, Sep 12, 2017 at 5:02 AM, Uwe Geercken <uwe.geerc...@web.de> wrote:
> Hello,
>
> I was wondering why if the QueryDatabaseTable processor creates internally
> an Avro schema, why is this schema not available as an attribute or saved to
> the registry?
>
> If it would, then one could reuse the schema. E.g. if I use the
> ConvertRecord processor and I specify an AvroReader as RecordReader, then
> this Reader will take the schema from the flowfile the QueryDatabaseTable
> processor creates. But the RecordWriter in the ConvertRecord - in my example
> a CSVRecordSetWriter requires the schema as an attribute or as a reference
> to the schema registry.
>
> I can see there is an ExtractAvroSchema processor but I don'see there is a
> way of combining the metadata into e.g. the ConvertRecord processor.
>
> Any help or ideas?
>
> Rgds,
>
> Uwe
I had a similar expectation when I was using QueryDatabaseTable or any
other processor creating Avro FlowFile which has its schema embedded,
combining new record reader/writer controllers.
Now, NiFi has "Inherit Record Schema" option as "Schema Access
Strategy" of RecordWriter, already merged in master branch.
https://issues.apache.org/jira/browse/NIFI-3921
I was able to reuse the Avro schema at subsequent flow using "Inherit
Record Schema", it's really useful. You can construct a flow like
below:
- QueryDatabaseTable
- outputs FlowFile with Avro schema embedded
- ConvertRecord
- AvroReader:
- "Schema Access Strategy" = "Use Embedded Avro Schema"
- CSVRecordSetWriter:
- "Schema Access Strategy" = "Inherit Record Schema"
- "Schema Write Strategy" = "Set 'avro.schema' Attribute"
This way, you don't have to have the schema in registry, and result
CSV FlowFile has 'avro.schema' attribute inheriting the one created by
QueryDatabaseTable.
Hope this helps.
Thanks,
Koji
On Tue, Sep 12, 2017 at 5:02 AM, Uwe Geercken <uwe.geerc...@web.de> wrote:
> Hello,
>
> I was wondering why if the QueryDatabaseTable processor creates internally
> an Avro schema, why is this schema not available as an attribute or saved to
> the registry?
>
> If it would, then one could reuse the schema. E.g. if I use the
> ConvertRecord processor and I specify an AvroReader as RecordReader, then
> this Reader will take the schema from the flowfile the QueryDatabaseTable
> processor creates. But the RecordWriter in the ConvertRecord - in my example
> a CSVRecordSetWriter requires the schema as an attribute or as a reference
> to the schema registry.
>
> I can see there is an ExtractAvroSchema processor but I don'see there is a
> way of combining the metadata into e.g. the ConvertRecord processor.
>
> Any help or ideas?
>
> Rgds,
>
> Uwe
