Well, not really. The Avro Schema Registry does perform validation of the Avro Schema. Which is something that a 'generic blob store' type of mechanism would not really provide. I think I recant my recommendation of making it more generic. There are definitely benefits to it being more specific. The ability to have an advanced UI that can make it easier to test the Jolt spec, as well as performing Jolt-specific validation probably makes it more beneficial to have a specific Jolt-oriented service. Additionally, it helps the user by narrowing down which service can be chosen. If it were more generic, the user may have many of these different services or many key/value pairs in one service that do not apply. By tying it to Jolt, it narrows down the user's choices to only those things that are applicable.
On Nov 20, 2019, at 10:49 AM, Etienne Jouvin <[email protected]<mailto:[email protected]>> wrote: Does it means that AvroSchemaRegistry is "not a good idea actually" ? I am pretty sure I misunderstood, because in this case there is a kind of compilation on schema. But you are right, the registry for JOLT specification is just a storage of blob. Le mer. 20 nov. 2019 à 16:36, Mark Payne <[email protected]<mailto:[email protected]>> a écrit : I would recommend that we also be careful about the naming here and tying this to Jolt. Really, this is just a mechanism for externalizing a big blob of text (or bytes). There are several other processors and controller services that do this, such as scripted components, Hadoop related processors that need things like core-site.xml, etc. It may be advantageous to consider this as a more generic way to access any such resource. A simple implementation would be purely configured through the UI but there could be other future implementations that are based on fetching from remote services, etc. Thanks -Mark Sent from my iPhone On Nov 20, 2019, at 10:28 AM, Joe Witt <[email protected]<mailto:[email protected]>> wrote: Yeah filing a JIRA would be good. Contributing a PR for it would be even better. It should have no impact on the schema registry controller service. This is different. Thanks On Wed, Nov 20, 2019 at 10:26 AM Etienne Jouvin <[email protected]<mailto:[email protected]>> wrote: Yes it would be a ControllerService as you described. There is currently three implementation : * AvroSchemaRegistry * ConfluentSchemaRegistry * HortonworksSchemaRegistry It could be based on something like them. May be I should send something on Jira or somewhere else to submit the idea to NiFi developers ? But it also means that the processor JoltTransformJSON and JoltTransformRecord should be changed. Le mer. 20 nov. 2019 à 16:08, Joe Witt <[email protected]<mailto:[email protected]>> a écrit : Hello Is the idea to have a place to store Jolt specifications that you could then access in various components? If so a simple ControllerService such as 'JoltSpecControllerService' which has a list of keys (names of specs) and values (the spec) would probably do the trick. Thanks On Wed, Nov 20, 2019 at 10:04 AM Otto Fowler <[email protected]<mailto:[email protected]>> wrote: I think that is a great idea, I’d suggest the same thing for protobuf specs as well. Even if the first step is the registry supporting raw bytes access and support…. On November 20, 2019 at 09:28:23, Etienne Jouvin ([email protected]<mailto:[email protected]>) wrote: Hello all. For reader and writers, there is the possibility to store the schema inside a schema registry. What do you think about having this type of mechanism for JolftTransformation ? Currently, I can put Jolt specification in variables and get them from it, but I think it could be nice tohave same as schema registry. Regards. Etienne Jouvin
