Re: Influence about removing RequiresInstanceClassLoading from AbstractHadoopProcessor processor

[Jeff]

By Hai Luo,

I haven't personally attempted to use NiFi with Hadoop in the way you
described.  If your Hadoop clusters are not secured I don't think you'll
have issues with UGI.  I wouldn't recommend running NiFi and Hadoop
clusters unsecured, however.

On Tue, Nov 12, 2019 at 8:40 PM abellnotring  wrote:

> Hi,jeff,
> There is no Kerberos authentication  in my HADOOP clusters , but I
> find UGI is initialized with an ExtendConfiguration(extend from hadoop
> Configruation) when those processor's instance was first scheduled .I would
> use nifi to connect different HADOOP cluster, will it  run into any
> issues?(I’m running tests for this)
>
>
> Thanks
>  By Hai Luo
> On 11/13/2019 07:01，Jeff  wrote：
>
> If you remove the @RequiresInstanceClassloading, the UserGroupInformation
> class from Hadoop (hadoop-common, if I remember correctly) will be shared
> across all instances that come from a particular NAR (such as PutHDFS,
> ListHDFS, FetchHDFS, etc, from nifi-hadoop-nar-x.y.z.nar).  If you are
> using Kerberos in those processors and configured different principals
> across the various processors, you could run into issues when the
> processors attempt to acquire new TGTs, most likely the first time a
> relogin is attempted.  UGI has some static state and
> @RequiresInstanceClassloading makes sure each instance of a processor with
> that annotation has its own classloader to keep that kind of state from
> being shared across instances.
>
> On Mon, Nov 11, 2019 at 9:41 PM abellnotring 
> wrote:
>
>> Hi,Peter & All
>>  I’m using kylo to manage the nifi flow(called feed in kylo),and
>> there are 4200 instances(600+ instances extended from
>>  AbstractHadoopProcessor) in my nifi canvas,The NIFI Non-Heap Memory has
>> increased more than  6GB after some days running ,which is extremely
>> abnormal . I have analyzed the class loaded into Compressed Class Space
>> ,and found most of the CCS was used by classes related by
>> AbstractHadoopProcessor.
>>So  I think removing RequiresInstanceClassLoading from
>> AbstractHadoopProcessor processor may be a Solution for reducing the CCS
>> used.
>>Do you have any ideas for this ?
>>
>>
>> 
>>
>>
>> 
>>
>>
>> 
>> 
>>   Thanks
>>
>>
>> By Hai Luo
>> On 11/12/2019 02:17，Shawn Weeks
>>  wrote：
>>
>> I’m assuming your talking about the snappy problem. If you use compress
>> content prior to puthdfs you can compress with Snappy as it uses the Java
>> Native Snappy Lib. The HDFS processors are limited to the actual Hadoop
>> Libraries so they’d have to change from Native to get around this. I’m
>> pretty sure we need instance loading to handle the other issues mentioned.
>>
>>
>>
>> Thanks
>>
>> Shawn
>>
>>
>>
>> *From: *Joe Witt 
>> *Reply-To: *"users@nifi.apache.org" 
>> *Date: *Monday, November 11, 2019 at 8:56 AM
>> *To: *"users@nifi.apache.org" 
>> *Subject: *Re: Influence about removing RequiresInstanceClassLoading
>> from AbstractHadoopProcessor processor
>>
>>
>>
>> Peter
>>
>>
>>
>> The most common challenge is if two isolated instances both want to use a
>> native lib.  No two native libs with the same name can be in the same jvm.
>> We need to solve that for sure.
>>
>>
>>
>> Thanks
>>
>>
>>
>> On Mon, Nov 11, 2019 at 9:53 AM Peter Turcsanyi 
>> wrote:
>>
>> Hi Hai Luo,
>>
>>
>>
>> @RequiresInstanceClassLoading makes possible to configure separate /
>> isolated "Additional Classpath Resources" settings on your HDFS processors
>> (eg. S3 storage driver on one of your PutHDFS and Azure Blob on the other).
>>
>>
>>
>> Is there any specific reason / use case why you are considering to remove
>> it?
>>
>>
>>
>> Regards,
>>
>> Peter Turcsanyi
>>
>>
>>
>> On Mon, Nov 11, 2019 at 3:30 PM abellnotring 
>> wrote:
>>
>> Hi,all
>>
>>  I’m considering removing the RequiresInstanceClassLoading annotation
>> from class AbstractHadoopProcessor,
>>
>>  Does anybody know the potential Influence?
>>
>>
>>
>> Thanks
>>
>> By Hai Luo
>>
>>

Re: Global Variable in NIFI

[KhajaAsmath Mohammed]

Thanks. You were right . 

Sent from my iPhone

> On Nov 20, 2019, at 6:10 PM, Bimal Mehta  wrote:
> 
> Click on the Process Group and open Variable Registry. Most likely it's 
> defined as Process Group variable inside Variable Registry
> 
>> On Wed, Nov 20, 2019, 1:24 PM KhajaAsmath Mohammed  
>> wrote:
>> Hi,
>> 
>> I have a flow in NIFI where I am not able to find the value of attribute 
>> that is being read from. I checked all the processors but didnt find this 
>> attribute. Looks like it is being read somewhere from globally. May I know 
>> how to do this?
>> 
>> 
>> 
>> 
>> 
>> Any idea where this property can be located?
>> 
>> Thanks,
>> Asmath

Re: How to use a Registry Variable in UpdateAttribute

[James McMahon]

That was it. Such a careless oversight. Thanks very much Mark.

On Wed, Nov 20, 2019 at 1:51 PM Mark Payne  wrote:

> Hi,
>
> The problem here, I think is that you're attempting to call `toUpper()` as
> a function that takes an argument. You'd need to change it instead to:
>
> ${filename:toUpper():substringAfter( ${myTokens:toUpper()} )}
>
> Thanks
> -Mark
>
>
> > On Nov 20, 2019, at 1:18 PM, James McMahon  wrote:
> >
> > I have an incoming filename value of LarryCurlyMoe_ABCDstooges.
> > I want to apply an expression in an UpdateAttribute processor that
> performs a substringAfter with a Registry variable reference.
> > My Registry variable myTokens has value ABCD.
> > I'd like to apply
> ${filename:toUpper():substringAfter(toUpper(${myTokens}))} to give me a new
> filename of STOOGES.
> > The expression above throws an error. How do I craft that expression to
> reference the value in Registry variable myTokens?
> > Thanks in advance.
>
>

Re: How to use a Registry Variable in UpdateAttribute

[Mark Payne]

Hi,

The problem here, I think is that you're attempting to call `toUpper()` as a 
function that takes an argument. You'd need to change it instead to:

${filename:toUpper():substringAfter( ${myTokens:toUpper()} )}

Thanks
-Mark


> On Nov 20, 2019, at 1:18 PM, James McMahon  wrote:
> 
> I have an incoming filename value of LarryCurlyMoe_ABCDstooges.
> I want to apply an expression in an UpdateAttribute processor that performs a 
> substringAfter with a Registry variable reference.
> My Registry variable myTokens has value ABCD.
> I'd like to apply ${filename:toUpper():substringAfter(toUpper(${myTokens}))} 
> to give me a new filename of STOOGES.
> The expression above throws an error. How do I craft that expression to 
> reference the value in Registry variable myTokens?
> Thanks in advance.

How to use a Registry Variable in UpdateAttribute

[James McMahon]

I have an incoming filename value of LarryCurlyMoe_ABCDstooges.
I want to apply an expression in an UpdateAttribute processor that performs
a substringAfter with a Registry variable reference.
My Registry variable myTokens has value ABCD.
I'd like to apply
${filename:toUpper():substringAfter(toUpper(${myTokens}))} to give me a new
filename of STOOGES.
The expression above throws an error. How do I craft that expression to
reference the value in Registry variable myTokens?
Thanks in advance.

Re: Jolt specification registry

[Pierre Villard]

A while back, I suggested (there is a JIRA for it) something similar for
Grok patterns (being able to have a global CS managing the grok patterns
instead of relying on local files to be distributed on the NiFi nodes). Not
the same but wanted to mention it in case it would drive towards something
generic.

Pierre

Le mer. 20 nov. 2019 à 16:54, Mark Payne  a écrit :

> Well, not really. The Avro Schema Registry does perform validation of the
> Avro Schema. Which is something that a 'generic blob store' type of
> mechanism would not really provide. I think I recant my recommendation of
> making it more generic. There are definitely benefits to it being more
> specific. The ability to have an advanced UI that can make it easier to
> test the Jolt spec, as well as performing Jolt-specific validation probably
> makes it more beneficial to have a specific Jolt-oriented service.
> Additionally, it helps the user by narrowing down which service can be
> chosen. If it were more generic, the user may have many of these different
> services or many key/value pairs in one service that do not apply. By tying
> it to Jolt, it narrows down the user's choices to only those things that
> are applicable.
>
> On Nov 20, 2019, at 10:49 AM, Etienne Jouvin 
> wrote:
>
> Does it means that AvroSchemaRegistry is "not a good idea actually" ?
>
> I am pretty sure I misunderstood, because in this case there is a kind of
> compilation on schema.
>
> But you are right, the registry for JOLT specification is just a storage
> of blob.
>
> Le mer. 20 nov. 2019 à 16:36, Mark Payne  a écrit :
>
>> I would recommend that we also be careful about the naming here and tying
>> this to Jolt. Really, this is just a mechanism for externalizing a big blob
>> of text (or bytes). There are several other processors and controller
>> services that do this, such as scripted components, Hadoop related
>> processors that need things like core-site.xml, etc.
>>
>> It may be advantageous to consider this as a more generic way to access
>> any such resource. A simple implementation would be purely configured
>> through the UI but there could be other future implementations that are
>> based on fetching from remote services, etc.
>>
>> Thanks
>> -Mark
>>
>> Sent from my iPhone
>>
>> On Nov 20, 2019, at 10:28 AM, Joe Witt  wrote:
>>
>> 
>> Yeah filing a JIRA would be good.  Contributing a PR for it would be even
>> better.  It should have no impact on the schema registry controller
>> service.  This is different.
>>
>> Thanks
>>
>> On Wed, Nov 20, 2019 at 10:26 AM Etienne Jouvin 
>> wrote:
>>
>>> Yes it would be a ControllerService as you described.
>>>
>>> There is currently three implementation :
>>> * AvroSchemaRegistry
>>> * ConfluentSchemaRegistry
>>> * HortonworksSchemaRegistry
>>>
>>> It could be based on something like them.
>>>
>>> May be I should send something on Jira or somewhere else to submit the
>>> idea to NiFi developers ?
>>>
>>> But it also means that the processor JoltTransformJSON and
>>> JoltTransformRecord should be changed.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Le mer. 20 nov. 2019 à 16:08, Joe Witt  a écrit :
>>>
 Hello

 Is the idea to have a place to store Jolt specifications that you could
 then access in various components?

 If so a simple ControllerService such as 'JoltSpecControllerService'
 which has a list of keys (names of specs) and values (the spec) would
 probably do the trick.

 Thanks

 On Wed, Nov 20, 2019 at 10:04 AM Otto Fowler 
 wrote:

> I think that is a great idea, I’d suggest the same thing for protobuf
> specs as well.
>
> Even if the first step is the registry supporting raw bytes access and
> support….
>
>
>
>
> On November 20, 2019 at 09:28:23, Etienne Jouvin (
> lapinoujou...@gmail.com) wrote:
>
> Hello all.
>
>
> For reader and writers, there is the possibility to store the schema
> inside a schema registry.
> What do you think about having this type of mechanism for
> JolftTransformation ?
> Currently, I can put Jolt specification in variables and get them from
> it, but I think it could be nice tohave same as schema registry.
>
> Regards.
>
> Etienne Jouvin
>
>
>
>
>

Re: Jolt specification registry

[Mark Payne]

Well, not really. The Avro Schema Registry does perform validation of the Avro 
Schema. Which is something that a 'generic blob store' type of mechanism would 
not really provide. I think I recant my recommendation of making it more 
generic. There are definitely benefits to it being more specific. The ability 
to have an advanced UI that can make it easier to test the Jolt spec, as well 
as performing Jolt-specific validation probably makes it more beneficial to 
have a specific Jolt-oriented service. Additionally, it helps the user by 
narrowing down which service can be chosen. If it were more generic, the user 
may have many of these different services or many key/value pairs in one 
service that do not apply. By tying it to Jolt, it narrows down the user's 
choices to only those things that are applicable.

On Nov 20, 2019, at 10:49 AM, Etienne Jouvin 
mailto:lapinoujou...@gmail.com>> wrote:

Does it means that AvroSchemaRegistry is "not a good idea actually" ?

I am pretty sure I misunderstood, because in this case there is a kind of 
compilation on schema.

But you are right, the registry for JOLT specification is just a storage of 
blob.

Le mer. 20 nov. 2019 à 16:36, Mark Payne 
mailto:marka...@hotmail.com>> a écrit :
I would recommend that we also be careful about the naming here and tying this 
to Jolt. Really, this is just a mechanism for externalizing a big blob of text 
(or bytes). There are several other processors and controller services that do 
this, such as scripted components, Hadoop related processors that need things 
like core-site.xml, etc.

It may be advantageous to consider this as a more generic way to access any 
such resource. A simple implementation would be purely configured through the 
UI but there could be other future implementations that are based on fetching 
from remote services, etc.

Thanks
-Mark

Sent from my iPhone

On Nov 20, 2019, at 10:28 AM, Joe Witt 
mailto:joe.w...@gmail.com>> wrote:


Yeah filing a JIRA would be good.  Contributing a PR for it would be even 
better.  It should have no impact on the schema registry controller service.  
This is different.

Thanks

On Wed, Nov 20, 2019 at 10:26 AM Etienne Jouvin 
mailto:lapinoujou...@gmail.com>> wrote:
Yes it would be a ControllerService as you described.

There is currently three implementation :
* AvroSchemaRegistry
* ConfluentSchemaRegistry
* HortonworksSchemaRegistry

It could be based on something like them.

May be I should send something on Jira or somewhere else to submit the idea to 
NiFi developers ?

But it also means that the processor JoltTransformJSON and JoltTransformRecord 
should be changed.







Le mer. 20 nov. 2019 à 16:08, Joe Witt 
mailto:joe.w...@gmail.com>> a écrit :
Hello

Is the idea to have a place to store Jolt specifications that you could then 
access in various components?

If so a simple ControllerService such as 'JoltSpecControllerService' which has 
a list of keys (names of specs) and values (the spec) would probably do the 
trick.

Thanks

On Wed, Nov 20, 2019 at 10:04 AM Otto Fowler 
mailto:ottobackwa...@gmail.com>> wrote:

I think that is a great idea, I’d suggest the same thing for protobuf specs as 
well.

Even if the first step is the registry supporting raw bytes access and support….





On November 20, 2019 at 09:28:23, Etienne Jouvin 
(lapinoujou...@gmail.com) wrote:

Hello all.


For reader and writers, there is the possibility to store the schema inside a 
schema registry.
What do you think about having this type of mechanism for JolftTransformation ?
Currently, I can put Jolt specification in variables and get them from it, but 
I think it could be nice tohave same as schema registry.

Regards.

Etienne Jouvin

Re: Jolt specification registry

[Mark Payne]

Fair enough. I forgot that we had the advanced UI for the Jolt Spec. That would 
definitely be beneficial.

On Nov 20, 2019, at 10:47 AM, Joe Witt 
mailto:joe.w...@gmail.com>> wrote:

...specific or generic - tradeoffs.  A specific service would allow Jolt Spec 
editing to have context and meaning.  A generic service for just blobs would 
not.  I think our Jolt stuff has an advanced UI that makes writing the specs a 
bit easier.  If true a more specific service would make sense. for this 
case anyway.

On Wed, Nov 20, 2019 at 10:36 AM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
I would recommend that we also be careful about the naming here and tying this 
to Jolt. Really, this is just a mechanism for externalizing a big blob of text 
(or bytes). There are several other processors and controller services that do 
this, such as scripted components, Hadoop related processors that need things 
like core-site.xml, etc.

It may be advantageous to consider this as a more generic way to access any 
such resource. A simple implementation would be purely configured through the 
UI but there could be other future implementations that are based on fetching 
from remote services, etc.

Thanks
-Mark

Sent from my iPhone

On Nov 20, 2019, at 10:28 AM, Joe Witt 
mailto:joe.w...@gmail.com>> wrote:


Yeah filing a JIRA would be good.  Contributing a PR for it would be even 
better.  It should have no impact on the schema registry controller service.  
This is different.

Thanks

On Wed, Nov 20, 2019 at 10:26 AM Etienne Jouvin 
mailto:lapinoujou...@gmail.com>> wrote:
Yes it would be a ControllerService as you described.

There is currently three implementation :
* AvroSchemaRegistry
* ConfluentSchemaRegistry
* HortonworksSchemaRegistry

It could be based on something like them.

May be I should send something on Jira or somewhere else to submit the idea to 
NiFi developers ?

But it also means that the processor JoltTransformJSON and JoltTransformRecord 
should be changed.







Le mer. 20 nov. 2019 à 16:08, Joe Witt 
mailto:joe.w...@gmail.com>> a écrit :
Hello

Is the idea to have a place to store Jolt specifications that you could then 
access in various components?

If so a simple ControllerService such as 'JoltSpecControllerService' which has 
a list of keys (names of specs) and values (the spec) would probably do the 
trick.

Thanks

On Wed, Nov 20, 2019 at 10:04 AM Otto Fowler 
mailto:ottobackwa...@gmail.com>> wrote:

I think that is a great idea, I’d suggest the same thing for protobuf specs as 
well.

Even if the first step is the registry supporting raw bytes access and support….





On November 20, 2019 at 09:28:23, Etienne Jouvin 
(lapinoujou...@gmail.com) wrote:

Hello all.


For reader and writers, there is the possibility to store the schema inside a 
schema registry.
What do you think about having this type of mechanism for JolftTransformation ?
Currently, I can put Jolt specification in variables and get them from it, but 
I think it could be nice tohave same as schema registry.

Regards.

Etienne Jouvin

Re: Jolt specification registry

[Etienne Jouvin]

Does it means that AvroSchemaRegistry is "not a good idea actually" ?

I am pretty sure I misunderstood, because in this case there is a kind of
compilation on schema.

But you are right, the registry for JOLT specification is just a storage of
blob.

Le mer. 20 nov. 2019 à 16:36, Mark Payne  a écrit :

> I would recommend that we also be careful about the naming here and tying
> this to Jolt. Really, this is just a mechanism for externalizing a big blob
> of text (or bytes). There are several other processors and controller
> services that do this, such as scripted components, Hadoop related
> processors that need things like core-site.xml, etc.
>
> It may be advantageous to consider this as a more generic way to access
> any such resource. A simple implementation would be purely configured
> through the UI but there could be other future implementations that are
> based on fetching from remote services, etc.
>
> Thanks
> -Mark
>
> Sent from my iPhone
>
> On Nov 20, 2019, at 10:28 AM, Joe Witt  wrote:
>
> 
> Yeah filing a JIRA would be good.  Contributing a PR for it would be even
> better.  It should have no impact on the schema registry controller
> service.  This is different.
>
> Thanks
>
> On Wed, Nov 20, 2019 at 10:26 AM Etienne Jouvin 
> wrote:
>
>> Yes it would be a ControllerService as you described.
>>
>> There is currently three implementation :
>> * AvroSchemaRegistry
>> * ConfluentSchemaRegistry
>> * HortonworksSchemaRegistry
>>
>> It could be based on something like them.
>>
>> May be I should send something on Jira or somewhere else to submit the
>> idea to NiFi developers ?
>>
>> But it also means that the processor JoltTransformJSON and
>> JoltTransformRecord should be changed.
>>
>>
>>
>>
>>
>>
>>
>> Le mer. 20 nov. 2019 à 16:08, Joe Witt  a écrit :
>>
>>> Hello
>>>
>>> Is the idea to have a place to store Jolt specifications that you could
>>> then access in various components?
>>>
>>> If so a simple ControllerService such as 'JoltSpecControllerService'
>>> which has a list of keys (names of specs) and values (the spec) would
>>> probably do the trick.
>>>
>>> Thanks
>>>
>>> On Wed, Nov 20, 2019 at 10:04 AM Otto Fowler 
>>> wrote:
>>>
 I think that is a great idea, I’d suggest the same thing for protobuf
 specs as well.

 Even if the first step is the registry supporting raw bytes access and
 support….




 On November 20, 2019 at 09:28:23, Etienne Jouvin (
 lapinoujou...@gmail.com) wrote:

 Hello all.


 For reader and writers, there is the possibility to store the schema
 inside a schema registry.
 What do you think about having this type of mechanism for
 JolftTransformation ?
 Currently, I can put Jolt specification in variables and get them from
 it, but I think it could be nice tohave same as schema registry.

 Regards.

 Etienne Jouvin

Re: Jolt specification registry

[Joe Witt]

...specific or generic - tradeoffs.  A specific service would allow Jolt
Spec editing to have context and meaning.  A generic service for just blobs
would not.  I think our Jolt stuff has an advanced UI that makes writing
the specs a bit easier.  If true a more specific service would make sense.
for this case anyway.

On Wed, Nov 20, 2019 at 10:36 AM Mark Payne  wrote:

> I would recommend that we also be careful about the naming here and tying
> this to Jolt. Really, this is just a mechanism for externalizing a big blob
> of text (or bytes). There are several other processors and controller
> services that do this, such as scripted components, Hadoop related
> processors that need things like core-site.xml, etc.
>
> It may be advantageous to consider this as a more generic way to access
> any such resource. A simple implementation would be purely configured
> through the UI but there could be other future implementations that are
> based on fetching from remote services, etc.
>
> Thanks
> -Mark
>
> Sent from my iPhone
>
> On Nov 20, 2019, at 10:28 AM, Joe Witt  wrote:
>
> 
> Yeah filing a JIRA would be good.  Contributing a PR for it would be even
> better.  It should have no impact on the schema registry controller
> service.  This is different.
>
> Thanks
>
> On Wed, Nov 20, 2019 at 10:26 AM Etienne Jouvin 
> wrote:
>
>> Yes it would be a ControllerService as you described.
>>
>> There is currently three implementation :
>> * AvroSchemaRegistry
>> * ConfluentSchemaRegistry
>> * HortonworksSchemaRegistry
>>
>> It could be based on something like them.
>>
>> May be I should send something on Jira or somewhere else to submit the
>> idea to NiFi developers ?
>>
>> But it also means that the processor JoltTransformJSON and
>> JoltTransformRecord should be changed.
>>
>>
>>
>>
>>
>>
>>
>> Le mer. 20 nov. 2019 à 16:08, Joe Witt  a écrit :
>>
>>> Hello
>>>
>>> Is the idea to have a place to store Jolt specifications that you could
>>> then access in various components?
>>>
>>> If so a simple ControllerService such as 'JoltSpecControllerService'
>>> which has a list of keys (names of specs) and values (the spec) would
>>> probably do the trick.
>>>
>>> Thanks
>>>
>>> On Wed, Nov 20, 2019 at 10:04 AM Otto Fowler 
>>> wrote:
>>>
 I think that is a great idea, I’d suggest the same thing for protobuf
 specs as well.

 Even if the first step is the registry supporting raw bytes access and
 support….




 On November 20, 2019 at 09:28:23, Etienne Jouvin (
 lapinoujou...@gmail.com) wrote:

 Hello all.


 For reader and writers, there is the possibility to store the schema
 inside a schema registry.
 What do you think about having this type of mechanism for
 JolftTransformation ?
 Currently, I can put Jolt specification in variables and get them from
 it, but I think it could be nice tohave same as schema registry.

 Regards.

 Etienne Jouvin

Re: Jolt specification registry

[Mark Payne]

I would recommend that we also be careful about the naming here and tying this 
to Jolt. Really, this is just a mechanism for externalizing a big blob of text 
(or bytes). There are several other processors and controller services that do 
this, such as scripted components, Hadoop related processors that need things 
like core-site.xml, etc.

It may be advantageous to consider this as a more generic way to access any 
such resource. A simple implementation would be purely configured through the 
UI but there could be other future implementations that are based on fetching 
from remote services, etc.

Thanks
-Mark

Sent from my iPhone

On Nov 20, 2019, at 10:28 AM, Joe Witt  wrote:


Yeah filing a JIRA would be good.  Contributing a PR for it would be even 
better.  It should have no impact on the schema registry controller service.  
This is different.

Thanks

On Wed, Nov 20, 2019 at 10:26 AM Etienne Jouvin 
mailto:lapinoujou...@gmail.com>> wrote:
Yes it would be a ControllerService as you described.

There is currently three implementation :
* AvroSchemaRegistry
* ConfluentSchemaRegistry
* HortonworksSchemaRegistry

It could be based on something like them.

May be I should send something on Jira or somewhere else to submit the idea to 
NiFi developers ?

But it also means that the processor JoltTransformJSON and JoltTransformRecord 
should be changed.







Le mer. 20 nov. 2019 à 16:08, Joe Witt 
mailto:joe.w...@gmail.com>> a écrit :
Hello

Is the idea to have a place to store Jolt specifications that you could then 
access in various components?

If so a simple ControllerService such as 'JoltSpecControllerService' which has 
a list of keys (names of specs) and values (the spec) would probably do the 
trick.

Thanks

On Wed, Nov 20, 2019 at 10:04 AM Otto Fowler 
mailto:ottobackwa...@gmail.com>> wrote:

I think that is a great idea, I’d suggest the same thing for protobuf specs as 
well.

Even if the first step is the registry supporting raw bytes access and support….




On November 20, 2019 at 09:28:23, Etienne Jouvin 
(lapinoujou...@gmail.com) wrote:

Hello all.


For reader and writers, there is the possibility to store the schema inside a 
schema registry.
What do you think about having this type of mechanism for JolftTransformation ?
Currently, I can put Jolt specification in variables and get them from it, but 
I think it could be nice tohave same as schema registry.

Regards.

Etienne Jouvin

Re: Jolt specification registry

[Etienne Jouvin]

For the PR...

If only I would have enough time for that ;)
Soon it will be end  year holidays, may be...



Le mer. 20 nov. 2019 à 16:28, Joe Witt  a écrit :

> Yeah filing a JIRA would be good.  Contributing a PR for it would be even
> better.  It should have no impact on the schema registry controller
> service.  This is different.
>
> Thanks
>
> On Wed, Nov 20, 2019 at 10:26 AM Etienne Jouvin 
> wrote:
>
>> Yes it would be a ControllerService as you described.
>>
>> There is currently three implementation :
>> * AvroSchemaRegistry
>> * ConfluentSchemaRegistry
>> * HortonworksSchemaRegistry
>>
>> It could be based on something like them.
>>
>> May be I should send something on Jira or somewhere else to submit the
>> idea to NiFi developers ?
>>
>> But it also means that the processor JoltTransformJSON and
>> JoltTransformRecord should be changed.
>>
>>
>>
>>
>>
>>
>>
>> Le mer. 20 nov. 2019 à 16:08, Joe Witt  a écrit :
>>
>>> Hello
>>>
>>> Is the idea to have a place to store Jolt specifications that you could
>>> then access in various components?
>>>
>>> If so a simple ControllerService such as 'JoltSpecControllerService'
>>> which has a list of keys (names of specs) and values (the spec) would
>>> probably do the trick.
>>>
>>> Thanks
>>>
>>> On Wed, Nov 20, 2019 at 10:04 AM Otto Fowler 
>>> wrote:
>>>
 I think that is a great idea, I’d suggest the same thing for protobuf
 specs as well.

 Even if the first step is the registry supporting raw bytes access and
 support….




 On November 20, 2019 at 09:28:23, Etienne Jouvin (
 lapinoujou...@gmail.com) wrote:

 Hello all.


 For reader and writers, there is the possibility to store the schema
 inside a schema registry.
 What do you think about having this type of mechanism for
 JolftTransformation ?
 Currently, I can put Jolt specification in variables and get them from
 it, but I think it could be nice tohave same as schema registry.

 Regards.

 Etienne Jouvin

Re: Jolt specification registry

[Etienne Jouvin]

Yes it would be a ControllerService as you described.

There is currently three implementation :
* AvroSchemaRegistry
* ConfluentSchemaRegistry
* HortonworksSchemaRegistry

It could be based on something like them.

May be I should send something on Jira or somewhere else to submit the idea
to NiFi developers ?

But it also means that the processor JoltTransformJSON and
JoltTransformRecord should be changed.







Le mer. 20 nov. 2019 à 16:08, Joe Witt  a écrit :

> Hello
>
> Is the idea to have a place to store Jolt specifications that you could
> then access in various components?
>
> If so a simple ControllerService such as 'JoltSpecControllerService' which
> has a list of keys (names of specs) and values (the spec) would probably do
> the trick.
>
> Thanks
>
> On Wed, Nov 20, 2019 at 10:04 AM Otto Fowler 
> wrote:
>
>> I think that is a great idea, I’d suggest the same thing for protobuf
>> specs as well.
>>
>> Even if the first step is the registry supporting raw bytes access and
>> support….
>>
>>
>>
>>
>> On November 20, 2019 at 09:28:23, Etienne Jouvin (lapinoujou...@gmail.com)
>> wrote:
>>
>> Hello all.
>>
>>
>> For reader and writers, there is the possibility to store the schema
>> inside a schema registry.
>> What do you think about having this type of mechanism for
>> JolftTransformation ?
>> Currently, I can put Jolt specification in variables and get them from
>> it, but I think it could be nice tohave same as schema registry.
>>
>> Regards.
>>
>> Etienne Jouvin
>>
>>
>>

Re: Jolt specification registry

[Joe Witt]

Hello

Is the idea to have a place to store Jolt specifications that you could
then access in various components?

If so a simple ControllerService such as 'JoltSpecControllerService' which
has a list of keys (names of specs) and values (the spec) would probably do
the trick.

Thanks

On Wed, Nov 20, 2019 at 10:04 AM Otto Fowler 
wrote:

> I think that is a great idea, I’d suggest the same thing for protobuf
> specs as well.
>
> Even if the first step is the registry supporting raw bytes access and
> support….
>
>
>
>
> On November 20, 2019 at 09:28:23, Etienne Jouvin (lapinoujou...@gmail.com)
> wrote:
>
> Hello all.
>
>
> For reader and writers, there is the possibility to store the schema
> inside a schema registry.
> What do you think about having this type of mechanism for
> JolftTransformation ?
> Currently, I can put Jolt specification in variables and get them from it,
> but I think it could be nice tohave same as schema registry.
>
> Regards.
>
> Etienne Jouvin
>
>
>

Re: Jolt specification registry

[Otto Fowler]

I think that is a great idea, I’d suggest the same thing for protobuf specs
as well.

Even if the first step is the registry supporting raw bytes access and
support….




On November 20, 2019 at 09:28:23, Etienne Jouvin (lapinoujou...@gmail.com)
wrote:

Hello all.


For reader and writers, there is the possibility to store the schema inside
a schema registry.
What do you think about having this type of mechanism for
JolftTransformation ?
Currently, I can put Jolt specification in variables and get them from it,
but I think it could be nice tohave same as schema registry.

Regards.

Etienne Jouvin

Jolt specification registry

[Etienne Jouvin]

Hello all.


For reader and writers, there is the possibility to store the schema inside
a schema registry.
What do you think about having this type of mechanism for
JolftTransformation ?
Currently, I can put Jolt specification in variables and get them from it,
but I think it could be nice tohave same as schema registry.

Regards.

Etienne Jouvin