Re: How to read a schema from a ScriptedReader
As a follow up, while implementing [1] I ran into [2] which is a
larger issue but affects the RecordSetWriters more than Readers as
they add more properties that are expected to come (in our case) from
the script. That makes things a bit more complicated, but I am still
looking into it. If you are coding a ScriptedReader you might be ok
with the guidance I gave earlier, but I have my doubts that trying to
use a ScriptedRecordSetWriter with a SchemaRegistry will work right
now. I'll post any findings in either/both [1] and [2], and as always
I welcome all comments, questions, and suggestions :)
Regards,
Matt
[1] https://issues.apache.org/jira/browse/NIFI-7343
[2] https://issues.apache.org/jira/browse/NIFI-5115
On Wed, Apr 8, 2020 at 8:19 PM Matt Burgess wrote:
>
> Jairo,
> We should probably move this to the dev list since we're getting into
> the NiFi API, but I wasn't sure you were subscribed, if you are let me
> know and we can move it over there.
>
> StandardConfigurationContext is in framework-core so you should not
> count on being able to use that class per se. Instead you should add a
> property like the one in SchemaAccessUtils:
>
> public static final PropertyDescriptor SCHEMA_REGISTRY = new
> PropertyDescriptor.Builder()
> .name("schema-registry")
> .displayName("Schema Registry")
> .description("Specifies the Controller Service to use for the
> Schema Registry")
> .identifiesControllerService(SchemaRegistry.class)
> .required(false)
> .build();
>
> Then from your code you can add that property in your overridden
> getSupportedPropertyDescriptors() method, and when you need it you can
> get the property from the context, but don't refer to it as a
> StandardConfigurationContext, there's an interface that has the same
> methods and is in the API that you have access to already:
>
> final SchemaRegistry schemaRegistry =
> context.getProperty(SCHEMA_REGISTRY).asControllerService(SchemaRegistry.class)
>
> Note that we're not getting the registry directly from the context,
> rather we're getting the property value from the context and then
> finding the corresponding SchemaRegistry, I believe that's done "under
> the hood" by the ControllerServiceLookup, the
> ".asControllerService(SchemaRegistry.class)" is the key part there.
>
> From there you can use the SchemaRegistry interface methods to get at
> the other things, for example:
>
> descriptor = getPropertyDescriptor(SCHEMA_ACCESS_STRATEGY.getName());
> schemaAccess = context.getProperty(descriptor).getValue();
> schemaAccessStrategy = getSchemaAccessStrategy(schemaAccess,
> schemaRegistry, context);
>
> Note that last method getSchemaAccessStrategy() is from
> SchemaAccessUtils which you don't currently have access to, so you
> might want to just copy all that over to your script. If you follow
> the calls and code from SchemaRegistryService, you should be able to
> bring any/all of that into your ScriptedReader so it can do the same
> kind of processing.
>
> Regards,
> Matt
>
> On Wed, Apr 8, 2020 at 7:03 PM Jairo Henao wrote:
> >
> > Thanks Matt,
> >
> > I have made some changes using AvroReader as an example, but I am still
> > able to access other ControllerServices.
> >
> > From the RecordReaderFactory I have access to a context that is of type
> > org.apache.nifi.controller.service.StandardConfigurationContext but with
> > this I don't know how to access the AvroSchemaRegistry.
> >
> > Do you know how I can access the ControllerServiceLookup?
> >
> >
> >
> >
> >
> >
> > On Wed, Apr 8, 2020 at 3:19 PM Matt Burgess wrote:
> >>
> >> Jairo,
> >>
> >> It is possible for a ScriptedReader to use a SchemaRegistry, although
> >> it isn't currently as easy as it could be. When the Scripted
> >> Reader/Writer components were added [1], much of the reusable code for
> >> schema registry processing was in the service implementation NAR, so
> >> the aforementioned Jira explains that using the SchemaRegistry
> >> interface is certainly possible in ScriptedReader but fairly
> >> cumbersome. Since then the SchemaRegistryService and associated util
> >> classes were moved to a nifi-avro-record-utils [2] so
> >> extensions/components can leverage schema registries. Unfortunately
> >> nifi-avro-record-utils is not currently included in the scripting NAR,
> >> but I have written up [3] to add this.
> >>
> >> In the meantime, take a look at SchemaRegistryService,
> >> SchemaAccessUtils, and an implementation class such as AvroReader, you
> >> could copy and paste all the necessary code to get your ScriptedReader
> >> to interact with a schema registry.
> >>
> >> Regards,
> >> Matt
> >>
> >> [1] https://issues.apache.org/jira/browse/NIFI-3734
> >> [2] https://issues.apache.org/jira/browse/NIFI-5123
> >> [3] https://issues.apache.org/jira/browse/NIFI-7343
> >>
> >> On Wed, Apr 8, 2020 at 2:01 PM Jairo Henao
> >> wrote:
> >> >
> >> > Hi all,
> >> >
> >> > Is there a way from a ScriptedReader (Controller
Re: How to read a schema from a ScriptedReader
Jairo,
We should probably move this to the dev list since we're getting into
the NiFi API, but I wasn't sure you were subscribed, if you are let me
know and we can move it over there.
StandardConfigurationContext is in framework-core so you should not
count on being able to use that class per se. Instead you should add a
property like the one in SchemaAccessUtils:
public static final PropertyDescriptor SCHEMA_REGISTRY = new
PropertyDescriptor.Builder()
.name("schema-registry")
.displayName("Schema Registry")
.description("Specifies the Controller Service to use for the
Schema Registry")
.identifiesControllerService(SchemaRegistry.class)
.required(false)
.build();
Then from your code you can add that property in your overridden
getSupportedPropertyDescriptors() method, and when you need it you can
get the property from the context, but don't refer to it as a
StandardConfigurationContext, there's an interface that has the same
methods and is in the API that you have access to already:
final SchemaRegistry schemaRegistry =
context.getProperty(SCHEMA_REGISTRY).asControllerService(SchemaRegistry.class)
Note that we're not getting the registry directly from the context,
rather we're getting the property value from the context and then
finding the corresponding SchemaRegistry, I believe that's done "under
the hood" by the ControllerServiceLookup, the
".asControllerService(SchemaRegistry.class)" is the key part there.
>From there you can use the SchemaRegistry interface methods to get at
the other things, for example:
descriptor = getPropertyDescriptor(SCHEMA_ACCESS_STRATEGY.getName());
schemaAccess = context.getProperty(descriptor).getValue();
schemaAccessStrategy = getSchemaAccessStrategy(schemaAccess,
schemaRegistry, context);
Note that last method getSchemaAccessStrategy() is from
SchemaAccessUtils which you don't currently have access to, so you
might want to just copy all that over to your script. If you follow
the calls and code from SchemaRegistryService, you should be able to
bring any/all of that into your ScriptedReader so it can do the same
kind of processing.
Regards,
Matt
On Wed, Apr 8, 2020 at 7:03 PM Jairo Henao wrote:
>
> Thanks Matt,
>
> I have made some changes using AvroReader as an example, but I am still able
> to access other ControllerServices.
>
> From the RecordReaderFactory I have access to a context that is of type
> org.apache.nifi.controller.service.StandardConfigurationContext but with this
> I don't know how to access the AvroSchemaRegistry.
>
> Do you know how I can access the ControllerServiceLookup?
>
>
>
>
>
>
> On Wed, Apr 8, 2020 at 3:19 PM Matt Burgess wrote:
>>
>> Jairo,
>>
>> It is possible for a ScriptedReader to use a SchemaRegistry, although
>> it isn't currently as easy as it could be. When the Scripted
>> Reader/Writer components were added [1], much of the reusable code for
>> schema registry processing was in the service implementation NAR, so
>> the aforementioned Jira explains that using the SchemaRegistry
>> interface is certainly possible in ScriptedReader but fairly
>> cumbersome. Since then the SchemaRegistryService and associated util
>> classes were moved to a nifi-avro-record-utils [2] so
>> extensions/components can leverage schema registries. Unfortunately
>> nifi-avro-record-utils is not currently included in the scripting NAR,
>> but I have written up [3] to add this.
>>
>> In the meantime, take a look at SchemaRegistryService,
>> SchemaAccessUtils, and an implementation class such as AvroReader, you
>> could copy and paste all the necessary code to get your ScriptedReader
>> to interact with a schema registry.
>>
>> Regards,
>> Matt
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-3734
>> [2] https://issues.apache.org/jira/browse/NIFI-5123
>> [3] https://issues.apache.org/jira/browse/NIFI-7343
>>
>> On Wed, Apr 8, 2020 at 2:01 PM Jairo Henao wrote:
>> >
>> > Hi all,
>> >
>> > Is there a way from a ScriptedReader (Controller Service) to read a schema
>> > that is stored in an AvroSchemaRegistry?
>> >
>> > I am using Groovy as a language.
>> >
>> > Additionally, how can I read additional properties that I have added to my
>> > ControllerService from Groovy code.
>> >
>> > Thanks in advance
>> >
>> >
>> > --
>> > Saludos
>> >
>> > Jairo Henao
>
>
>
> --
> Saludos
>
> Jairo Henao
>
> Chat Skype: jairo.henao.05
>
Re: How to read a schema from a ScriptedReader
Thanks Matt, I have made some changes using AvroReader as an example, but I am still able to access other ControllerServices. >From the RecordReaderFactory I have access to a context that is of type *org.apache.nifi.controller.service.StandardConfigurationContext* but with this I don't know how to access the AvroSchemaRegistry. Do you know how I can access the ControllerServiceLookup? On Wed, Apr 8, 2020 at 3:19 PM Matt Burgess wrote: > Jairo, > > It is possible for a ScriptedReader to use a SchemaRegistry, although > it isn't currently as easy as it could be. When the Scripted > Reader/Writer components were added [1], much of the reusable code for > schema registry processing was in the service implementation NAR, so > the aforementioned Jira explains that using the SchemaRegistry > interface is certainly possible in ScriptedReader but fairly > cumbersome. Since then the SchemaRegistryService and associated util > classes were moved to a nifi-avro-record-utils [2] so > extensions/components can leverage schema registries. Unfortunately > nifi-avro-record-utils is not currently included in the scripting NAR, > but I have written up [3] to add this. > > In the meantime, take a look at SchemaRegistryService, > SchemaAccessUtils, and an implementation class such as AvroReader, you > could copy and paste all the necessary code to get your ScriptedReader > to interact with a schema registry. > > Regards, > Matt > > [1] https://issues.apache.org/jira/browse/NIFI-3734 > [2] https://issues.apache.org/jira/browse/NIFI-5123 > [3] https://issues.apache.org/jira/browse/NIFI-7343 > > On Wed, Apr 8, 2020 at 2:01 PM Jairo Henao > wrote: > > > > Hi all, > > > > Is there a way from a ScriptedReader (Controller Service) to read a > schema that is stored in an AvroSchemaRegistry? > > > > I am using Groovy as a language. > > > > Additionally, how can I read additional properties that I have added to > my ControllerService from Groovy code. > > > > Thanks in advance > > > > > > -- > > Saludos > > > > Jairo Henao > -- Saludos Jairo Henao *Chat Skype: jairo.henao.05*
Re: How to read a schema from a ScriptedReader
Jairo, It is possible for a ScriptedReader to use a SchemaRegistry, although it isn't currently as easy as it could be. When the Scripted Reader/Writer components were added [1], much of the reusable code for schema registry processing was in the service implementation NAR, so the aforementioned Jira explains that using the SchemaRegistry interface is certainly possible in ScriptedReader but fairly cumbersome. Since then the SchemaRegistryService and associated util classes were moved to a nifi-avro-record-utils [2] so extensions/components can leverage schema registries. Unfortunately nifi-avro-record-utils is not currently included in the scripting NAR, but I have written up [3] to add this. In the meantime, take a look at SchemaRegistryService, SchemaAccessUtils, and an implementation class such as AvroReader, you could copy and paste all the necessary code to get your ScriptedReader to interact with a schema registry. Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-3734 [2] https://issues.apache.org/jira/browse/NIFI-5123 [3] https://issues.apache.org/jira/browse/NIFI-7343 On Wed, Apr 8, 2020 at 2:01 PM Jairo Henao wrote: > > Hi all, > > Is there a way from a ScriptedReader (Controller Service) to read a schema > that is stored in an AvroSchemaRegistry? > > I am using Groovy as a language. > > Additionally, how can I read additional properties that I have added to my > ControllerService from Groovy code. > > Thanks in advance > > > -- > Saludos > > Jairo Henao
Not Seeing Provenance data
Hi, When I go to "View data provenance" in Nifi, I never see any logs for my flow. Am I missing some configuration setting somewhere? thanks, Darren
How to read a schema from a ScriptedReader
Hi all, Is there a way from a ScriptedReader (Controller Service) to read a schema that is stored in an AvroSchemaRegistry? I am using Groovy as a language. Additionally, how can I read additional properties that I have added to my ControllerService from Groovy code. Thanks in advance -- Saludos Jairo Henao
