Re: Jolt specification registry
I think the idea would be to provide API support for the schema registry where you could work with it as just versioned blobs, and then the level of support on top of that ( to the ui etc ) would be dependent on the implementation, with AVRO being the first concrete implementation. You could then “manage” grok patterns, jolt transforms, protobuf specs, and other things…. maybe even sql, ES … other queries. On November 20, 2019 at 11:55:05, Pierre Villard ( pierre.villard...@gmail.com) wrote: A while back, I suggested (there is a JIRA for it) something similar for Grok patterns (being able to have a global CS managing the grok patterns instead of relying on local files to be distributed on the NiFi nodes). Not the same but wanted to mention it in case it would drive towards something generic. Pierre Le mer. 20 nov. 2019 à 16:54, Mark Payne a écrit : > Well, not really. The Avro Schema Registry does perform validation of the > Avro Schema. Which is something that a 'generic blob store' type of > mechanism would not really provide. I think I recant my recommendation of > making it more generic. There are definitely benefits to it being more > specific. The ability to have an advanced UI that can make it easier to > test the Jolt spec, as well as performing Jolt-specific validation probably > makes it more beneficial to have a specific Jolt-oriented service. > Additionally, it helps the user by narrowing down which service can be > chosen. If it were more generic, the user may have many of these different > services or many key/value pairs in one service that do not apply. By tying > it to Jolt, it narrows down the user's choices to only those things that > are applicable. > > On Nov 20, 2019, at 10:49 AM, Etienne Jouvin > wrote: > > Does it means that AvroSchemaRegistry is "not a good idea actually" ? > > I am pretty sure I misunderstood, because in this case there is a kind of > compilation on schema. > > But you are right, the registry for JOLT specification is just a storage > of blob. > > Le mer. 20 nov. 2019 à 16:36, Mark Payne a écrit : > >> I would recommend that we also be careful about the naming here and tying >> this to Jolt. Really, this is just a mechanism for externalizing a big blob >> of text (or bytes). There are several other processors and controller >> services that do this, such as scripted components, Hadoop related >> processors that need things like core-site.xml, etc. >> >> It may be advantageous to consider this as a more generic way to access >> any such resource. A simple implementation would be purely configured >> through the UI but there could be other future implementations that are >> based on fetching from remote services, etc. >> >> Thanks >> -Mark >> >> Sent from my iPhone >> >> On Nov 20, 2019, at 10:28 AM, Joe Witt wrote: >> >> >> Yeah filing a JIRA would be good. Contributing a PR for it would be even >> better. It should have no impact on the schema registry controller >> service. This is different. >> >> Thanks >> >> On Wed, Nov 20, 2019 at 10:26 AM Etienne Jouvin >> wrote: >> >>> Yes it would be a ControllerService as you described. >>> >>> There is currently three implementation : >>> * AvroSchemaRegistry >>> * ConfluentSchemaRegistry >>> * HortonworksSchemaRegistry >>> >>> It could be based on something like them. >>> >>> May be I should send something on Jira or somewhere else to submit the >>> idea to NiFi developers ? >>> >>> But it also means that the processor JoltTransformJSON and >>> JoltTransformRecord should be changed. >>> >>> >>> >>> >>> >>> >>> >>> Le mer. 20 nov. 2019 à 16:08, Joe Witt a écrit : >>> Hello Is the idea to have a place to store Jolt specifications that you could then access in various components? If so a simple ControllerService such as 'JoltSpecControllerService' which has a list of keys (names of specs) and values (the spec) would probably do the trick. Thanks On Wed, Nov 20, 2019 at 10:04 AM Otto Fowler wrote: > I think that is a great idea, I’d suggest the same thing for protobuf > specs as well. > > Even if the first step is the registry supporting raw bytes access and > support…. > > > > > On November 20, 2019 at 09:28:23, Etienne Jouvin ( > lapinoujou...@gmail.com) wrote: > > Hello all. > > > For reader and writers, there is the possibility to store the schema > inside a schema registry. > What do you think about having this type of mechanism for > JolftTransformation ? > Currently, I can put Jolt specification in variables and get them from > it, but I think it could be nice tohave same as schema registry. > > Regards. > > Etienne Jouvin > > > > >
Re: PutKudu 1.10.0 Processor generates tons of log warnings - org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed session; this is unsafe
Thank you Pierre, your workaround works like a charm! JIRA request has been created: https://issues.apache.org/jira/projects/NIFI/issues/NIFI-6895 From: Pierre Villard Reply to: "users@nifi.apache.org" Date: Thursday, 21 November 2019 at 10:52 To: "users@nifi.apache.org" Subject: Re: PutKudu 1.10.0 Processor generates tons of log warnings - org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed session; this is unsafe Hi Joseph, Yes please file a JIRA, we need to have a look even though it might be due to a version change in the Kudu dependency. To workaround the issue, adding the below line into logback.xml should fix it: Hope this helps, Pierre Le jeu. 21 nov. 2019 à 10:33, mailto:josef.zahn...@swisscom.com>> a écrit : Hi guys We have just upgraded to NiFi 1.10.0 yesterday. We have seen that we got 1’000 times more logs (nifi-app.log) since the upgrade, caused mainly by the PutKudu processor. The logmessage we always get is: 2019-11-21 08:42:27,627 WARN [Timer-Driven Process Thread-2] org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed session; this is unsafe 2019-11-21 08:42:27,627 WARN [Timer-Driven Process Thread-40] org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed session; this is unsafe 2019-11-21 08:42:27,627 WARN [Timer-Driven Process Thread-2] org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed session; this is unsafe 2019-11-21 08:42:27,627 WARN [Timer-Driven Process Thread-40] org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed session; this is unsafe … The PutKudu processor itself seems to work fine. We have the same amount of inserts as before the upgrade to NiFi 1.10.0. The line of code which from Apache Kudu which reflects the message is here: (line 547) https://github.com/apache/kudu/blob/master/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java Questions: 1. How can I suppress just this single type of warning in NiFi? Most likely in logback.xml – but is there any tutorial how to do this? The messages generates multiple gigabytes of logs on our nodes per hour… we have to get rid of this kind of message as soon as possible. 2. Seems that from NiFi 1.9.2 to NiFi 1.10.0 something has changed in the PutKudu processor, we haven’t seen this message before. Shall I fill a jira ticket for that? Or has anybody an idea? Thanks in advance, Josef smime.p7s Description: S/MIME Cryptographic Signature
Re: PutKudu 1.10.0 Processor generates tons of log warnings - org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed session; this is unsafe
Hi Joseph, Yes please file a JIRA, we need to have a look even though it might be due to a version change in the Kudu dependency. To workaround the issue, adding the below line into logback.xml should fix it: Hope this helps, Pierre Le jeu. 21 nov. 2019 à 10:33, a écrit : > Hi guys > > > > We have just upgraded to NiFi *1.10.0* yesterday. We have seen that we > got 1’000 times more logs (nifi-app.log) since the upgrade, caused mainly > by the *PutKudu* processor. > > > > The logmessage we always get is: > > 2019-11-21 08:42:27,627 WARN [Timer-Driven Process Thread-2] > org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed > session; this is unsafe > > 2019-11-21 08:42:27,627 WARN [Timer-Driven Process Thread-40] > org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed > session; this is unsafe > > 2019-11-21 08:42:27,627 WARN [Timer-Driven Process Thread-2] > org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed > session; this is unsafe > > 2019-11-21 08:42:27,627 WARN [Timer-Driven Process Thread-40] > org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed > session; this is unsafe > > … > > > > The PutKudu processor itself seems to work fine. We have the same amount > of inserts as before the upgrade to NiFi 1.10.0. > > > > The line of code which from Apache Kudu which reflects the message is > here: (line 547) > > > https://github.com/apache/kudu/blob/master/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java > > > > *Questions:* > >1. How can I suppress just this single type of warning in NiFi? Most >likely in logback.xml – but is there any tutorial how to do this? The >messages generates multiple gigabytes of logs on our nodes per hour… we >have to get rid of this kind of message as soon as possible. >2. Seems that from NiFi 1.9.2 to NiFi 1.10.0 something has changed in >the PutKudu processor, we haven’t seen this message before. Shall I fill a >jira ticket for that? Or has anybody an idea? > > > > Thanks in advance, > > Josef > > >
PutKudu 1.10.0 Processor generates tons of log warnings - org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed session; this is unsafe
Hi guys We have just upgraded to NiFi 1.10.0 yesterday. We have seen that we got 1’000 times more logs (nifi-app.log) since the upgrade, caused mainly by the PutKudu processor. The logmessage we always get is: 2019-11-21 08:42:27,627 WARN [Timer-Driven Process Thread-2] org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed session; this is unsafe 2019-11-21 08:42:27,627 WARN [Timer-Driven Process Thread-40] org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed session; this is unsafe 2019-11-21 08:42:27,627 WARN [Timer-Driven Process Thread-2] org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed session; this is unsafe 2019-11-21 08:42:27,627 WARN [Timer-Driven Process Thread-40] org.apache.kudu.client.AsyncKuduSession Applying an operation in a closed session; this is unsafe … The PutKudu processor itself seems to work fine. We have the same amount of inserts as before the upgrade to NiFi 1.10.0. The line of code which from Apache Kudu which reflects the message is here: (line 547) https://github.com/apache/kudu/blob/master/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java Questions: 1. How can I suppress just this single type of warning in NiFi? Most likely in logback.xml – but is there any tutorial how to do this? The messages generates multiple gigabytes of logs on our nodes per hour… we have to get rid of this kind of message as soon as possible. 2. Seems that from NiFi 1.9.2 to NiFi 1.10.0 something has changed in the PutKudu processor, we haven’t seen this message before. Shall I fill a jira ticket for that? Or has anybody an idea? Thanks in advance, Josef smime.p7s Description: S/MIME Cryptographic Signature
