NioAsyncLoadBalanceClient: Unable to connect for load balancing
Hi Everyone, I am having errors in my Cluster when I try to create a new local connection: *ERROR [Load-Balanced Client Thread-2] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to ec2-nifi-node-002.internal:8443 for load balancingjava.net.ConnectException: Connection timed out at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:476) at java.base/sun.nio.ch.Net.connect(Net.java:468) at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:692) at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100) at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:84) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:495) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:438) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:233) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)* The cluster has 3 nodes, and always when trying to add a local connection, one of the 3 randomly disconnects from the Cluster. After it reconnects, it already brings the latest changes. I can add processors and process groups and it doesn't throw an error. This is how they are configured: nifi.cluster.protocol.heartbeat.interval=5 sec nifi.cluster.protocol.heartbeat.missable.max=8 nifi.cluster.protocol.is.secure=true # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address=ec2-nifi-node-002.internal nifi.cluster.node.protocol.port=8445 nifi.cluster.node.protocol.threads=10 nifi.cluster.node.protocol.max.threads=50 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=30 sec nifi.cluster.node.read.timeout=30 sec nifi.cluster.node.max.concurrent.requests=150 nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=2 mins nifi.cluster.flow.election.max.candidates=3 # cluster load balancing properties # nifi.cluster.load.balance.host= nifi.cluster.load.balance.port=6342 nifi.cluster.load.balance.connections.per.node=4 nifi.cluster.load.balance.max.thread.count=12 nifi.cluster.load.balance.comms.timeout=30 sec I am using NIFI-1.14.0 and Open JDK-11.0.12. Each node has a processor with 2 Cores and 5GB of memory. NIFI has 10700 processors, 2600 input ports, 2600 output ports, 0 remote process groups, 16500 connections and 3000 Process groups. We will have touched some limit? -- Jairo Henao @jairohenaorojas
Re: Default value for FlowFile Expiration
Thank you Lehel, As I need it at the level of everything in Nifi, I have configured it in the root PG. The only bad thing is that I have to fix what already exists manually. On Fri, Oct 15, 2021 at 10:45 PM Lehel Boér wrote: > Hi Jairo, > > You can define a default flowfile expiration time for a Process Group. > Right click on the Process Group -> Configure -> Default -> Default > FlowFile Expiration. > > Kind Regards, > Lehel Boér > > Jairo Henao ezt írta (időpont: 2021. okt. > 15., P, 18:41): > >> Hi Community, >> >> Is there any way to set a default value for the FlowFile Expiration >> defined in the connections?. >> >> What I want is to avoid queued FlowFiles indefinitely. >> >> Thanks, >> >> -- >> Jairo Henao >> @jairohenaorojas >> >> -- Jairo Henao @jairohenaorojas
Default value for FlowFile Expiration
Hi Community, Is there any way to set a default value for the FlowFile Expiration defined in the connections?. What I want is to avoid queued FlowFiles indefinitely. Thanks, -- Jairo Henao @jairohenaorojas
Better approach to run SQL and load it as attribute
Hi all, I need in several cases of my process to execute a SQL that counts and averages data from a table. This is a SQL with two columns and one row that I always need to load as an attribute, so I use: ExecuteSQL -> RouteOnAttribute (To check the executesql.row.count != 0) -> ConvertAvroTOJSON -> EvaluateJsonPath. I would like the simplest approach, is there a shorter one? Two things that would be good to improve/do: - ExecuteSQL should be configured so that it does not generate output flow if there are no results or that it has a new relationship (e.g. not data returned) - Create a new processor that does these steps, something like an ExecuteSQLToAttributes. Thanks, -- Jairo Henao @jairohenaorojas
Re: FetchSFTP: Rename file on move
Thanks for the suggestions. The combination of UpdateAttribute (Setting the new filename) with PutSFTP works fine! On Tue, Aug 11, 2020 at 5:59 PM Andy LoPresto wrote: > You can also use an UpdateAttribute processor to change the “filename” > attribute, which is what any “file persistence” processor (PutSFTP, > PutFile, etc.) will use when writing the file out. > > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com * > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Aug 11, 2020, at 3:47 PM, Joe Witt wrote: > > Jairo > > You can use a PutSFTP after Fetch to place it where you want. > > Thanks > > On Tue, Aug 11, 2020 at 3:16 PM Jairo Henao > wrote: > >> Hi community, >> >> Is there a way to rename a file before moving it with FetchSFTP? >> >> After processing a file, I need to move it to a folder and add a >> timestamp suffix to it. The file in the source always has the same name, but >> I need that when moving it they are not overwritten. >> >> Any ideas or is it necessary to request a modification to the processor? >> >> Thanks >> >> -- >> Jairo Henao >> @jairohenaorojas >> >> > -- Jairo Henao @jairohenaorojas
FetchSFTP: Rename file on move
Hi community, Is there a way to rename a file before moving it with FetchSFTP? After processing a file, I need to move it to a folder and add a timestamp suffix to it. The file in the source always has the same name, but I need that when moving it they are not overwritten. Any ideas or is it necessary to request a modification to the processor? Thanks -- Jairo Henao @jairohenaorojas
Re: Modifying JSON from flowfile attribute using JoltTransformJSON
Hi Bob,
The JoltTransformJSON processor operates on the content of the flowfile.
You can use attributes in your Jolt specification or even build Nifi
expressions, but you must ensure that the content of your flowfile contains
the JSON you are going to transform.
You could use a ReplaceText and thus pass the value of the attribute to the
content of the flowfile. So then apply the JoltTransform. Please note that
this will cause the entire content of your flowfile to be replaced.
On Wed, Jul 8, 2020 at 3:38 PM bsavard wrote:
> Hi,
>
> NiFi noobie here, so go easy on me :)
>
> I have a flowfile attribute (let's call it "Data") that contains JSON. I
> need to remove certain pairs from that JSON and then use the resulting JSON
> as the body of a POST to a URL.
>
> The flow file attribute "Data" contains the following JSON:
>
> [{
> "one": "a",
> "two": "b",
> "three": "c",
> "four": "d"
> }, {
> "one": "e",
> "two": "f",
> "three": "g",
> "four": "h"
> }]
>
> I want to remove elements "two" and "four", resulting in this JSON:
>
> [{
> "one": "a",
> "three": "c"
> }, {
> "one": "e",
> "three": "g"
> }]
>
> I'm trying to use the JoltTransformJSON processor.
> My Jolt Specification is this:
>
> [{
> "operation": "remove",
> "spec": {
> "*": {
> "two": "",
> "four": ""
> }
> }
> }]
>
> When I test it via the Advanced button UI, it works.
> But when I run the flow, I get an error:
> "com.bazaarvoice.jolt.exception.JsonUnmarshalException: Unable to unmarshal
> JSON to an Object"
>
> How exactly should I be referring to the flow file attribute in the spec?
> I've even tried a simple spec like this (where uuid is a flowfile attribute
> that contains "foo":
> [{
> "operation": "default",
> "spec": {
> "*": {
> "five": "${uuid}"
> }
> }
> }]
> Again, this works when I test it via the Advanced button UI, but I get the
> same error when I run the actual flow. It feels like there's some basic
> thing I'm missing.
> TIA for any pointers!
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
>
--
Saludos
Jairo Henao
*Chat Skype: jairo.henao.05*
Re: Not Seeing Provenance data
>>> >>>> -Mark >>>> >>>> >>>> >>>> [1] https://issues.apache.org/jira/browse/NIFI-7346 >>>> >>>> >>>> >>>> On Apr 9, 2020, at 11:14 AM, Dobbernack, Harald (Key-Work) < >>>> harald.dobbern...@key-work.de> wrote: >>>> >>>> >>>> >>>> Hi Mark, >>>> >>>> >>>> >>>> I can confirm after testing that if no provenance event has been >>>> generated in a time greater than the set >>>> nifi.provenance.repository.max.storage.time >>>> then as expected the last recorded provenance events don’t exist anymore >>>> but also from then on any new provenance events are also not searchable, >>>> the provenance Search remains completely empty regardless of how many flows >>>> are active. As described also *.prov file is then missing in provenance >>>> repository. After restart of Nifi new prov File will be generated and >>>> provenance will work again, but only showing stuff generated since last >>>> NiFi Start. >>>> >>>> >>>> >>>> So yes, I’d say your Idea >>>> >>>> ‘If so, then I think that would understand why it deleted the data. >>>> It’s trying to age off old data >>>> >>>> but unfortunately it doesn’t perform a check to first determine >>>> whether or not the “old file” >>>> >>>> that it’s about to delete is also the “active file”.’ >>>> >>>> fits very nicely to my test. >>>> >>>> >>>> >>>> As a workaround we’re going to set a greater >>>> nifi.provenance.repository.max.storage.time >>>> until this can be resolved. >>>> >>>> >>>> >>>> Thanks again for looking into this. >>>> >>>> Harald >>>> >>>> >>>> >>>> >>>> >>>> *Von:* Dobbernack, Harald (Key-Work) >>>> *Gesendet:* Donnerstag, 9. April 2020 15:22 >>>> *An:* users@nifi.apache.org >>>> *Betreff:* AW: Not Seeing Provenance data >>>> >>>> >>>> >>>> Hi Mark, >>>> >>>> >>>> >>>> thank you for looking into this. >>>> >>>> >>>> >>>> The nifi.provenance.repository.max.storage.time setting might explain >>>> why I haven’t been experiencing the effect so often since changing from the >>>> default to 120 hours a few months ago >>>> >>>> >>>> >>>> But I believe provenance stopped working last time although there was >>>> an ‘active’ flows in wait Processor, expiring every hour, going on to ‘send >>>> a message’ before being rerouted to the same wait processor. I would have >>>> expected this generates provenance entries? As I am not actually 100% sure >>>> if that wait processor was in use when last provenance got lost I will >>>> check with a testing system to see if I can reproduce provenance breakage >>>> when no active flows are around for a time greater >>>> nifi.provenance.repository.max.storage.time and I will get back to >>>> you. >>>> >>>> >>>> >>>> Thank you! >>>> >>>> Harald >>>> >>>> >>>> >>>> >>>> >>>> *Von:* Mark Payne >>>> *Gesendet:* Donnerstag, 9. April 2020 14:41 >>>> *An:* users@nifi.apache.org >>>> *Betreff:* Re: Not Seeing Provenance data >>>> >>>> >>>> >>>> Hey Daren, Herald, >>>> >>>> >>>> >>>> Thanks for the note. I have seen this once before but couldn’t figure >>>> out what caused it. Restarting addressed the issue. >>>> >>>> >>>> >>>> I think I may understand the problem, now, though, after looking at it >>>> again. >>>> >>>> >>>> >>>> In nifi.properties, there are a couple of property named >>>> “nifi.provenance.repository.max.storage.time” that defaults to “24 hours" >>>> >>>> Is it possible that you went 24 hours (or whatever value is set for >>>> that property) without generating any Provenance events? >>>> >>>> >>>> >>>> If so, then I think that would understand why it deleted the data. It’s >>>> trying to age off old data but unfortunately it doesn’t perform a check to >>>> first determine whether or not the “old file” that it’s about to delete is >>>> also the “active file”. >>>> >>>> >>>> >>>> Can you confirm whether or not you would expect to see 24 hours pass >>>> without any provenance data? >>>> >>>> >>>> >>>> Thanks >>>> >>>> -Mark >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Apr 9, 2020, at 4:32 AM, Dobbernack, Harald (Key-Work) < >>>> harald.dobbern...@key-work.de> wrote: >>>> >>>> >>>> >>>> What I noticed is that as long as provenance is working there will be >>>> *.prov files in the directory. When Provenance isn’t working these files >>>> are not to be seen. Maybe some Cleaning Process deletes those files >>>> prematurely or the process building them doesn’t work any more? >>>> >>>> >>>> >>>> *Von:* Dobbernack, Harald (Key-Work) >>>> *Gesendet:* Donnerstag, 9. April 2020 10:27 >>>> *An:* users@nifi.apache.org >>>> *Betreff:* AW: Not Seeing Provenance data >>>> >>>> >>>> >>>> This is something I experience too from time to time. My quick and >>>> dirty workaround is stop nifi, delete everything in the provenance >>>> directory, restart…. Then Provenance is usable again (of course only with >>>> data since the delete) . I’m hoping very much there is a better way, >>>> someone can show us better settings or a potential bug can be discovered… >>>> >>>> >>>> >>>> *Von:* Darren Govoni >>>> *Gesendet:* Mittwoch, 8. April 2020 20:31 >>>> *An:* users@nifi.apache.org >>>> *Betreff:* Not Seeing Provenance data >>>> >>>> >>>> >>>> Hi, >>>> >>>> When I go to "View data provenance" in Nifi, I never see any logs for >>>> my flow. Am I missing some configuration setting somewhere? >>>> >>>> >>>> >>>> thanks, >>>> >>>> Darren >>>> >>>> >>>> >>>> >>>> >>>> *Harald Dobbernack* >>>> Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany >>>> | https://www.key-work.de | Datenschutz >>>> <https://www.key-work.de/de/footer/datenschutz.html> >>>> Fon: +49-721-78203-264 | E-Mail: harald.dobbern...@key-work.de | Fax: >>>> +49-721-78203-10 >>>> >>>> Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim >>>> Geschäftsführer: Andreas Stappert, Tobin Wotring >>>> >>>> >>>> >>>> -- Saludos Jairo Henao *Chat Skype: jairo.henao.05*
Re: Extract the validation details in ValidateRecord
Thanks Cris, It would be great if the ValidateRecord had a configuration parameter where you define the name of the attribute to save the details. On Fri, May 1, 2020 at 10:12 AM Chris Sampson wrote: > Unfortunately, that's the only way currently with the existing > ValidateRecord processor - the details are added to the ROUTE Provenance > Event, which you can query via the REST API or (generally better) export > via the Site-to-Site Reporting Task. > > There was an PR opened a while back to make a change but it went stale > after comments were made. I know because we wanted to do the same thing for > our own project, so I made some customisations for our project directly > (effectively using parts of the aforementioned PRs, accepting the issues > raised in the PR comments). > > > *Chris Sampson* > IT Consultant > chris.samp...@naimuri.com > > > > On Fri, 1 May 2020 at 15:30, Jairo Henao > wrote: > >> Hi all, >> >> Is there a way to copy into an attribute the details of a validation when >> it fails using a ValidateRecord? >> >> The only thing I find is related to reporting tasks, which involve a >> series of extra steps. >> >> The only thing I need is to prepare an email with the details of the >> failure. >> >> Thanks, >> >> >> -- Saludos Jairo Henao *Chat Skype: jairo.henao.05*
Re: How to read a schema from a ScriptedReader
Thanks Matt,
Yes, I am subscribed to the development list.
I will review your comments. At the moment I am adding the schemas as
additional properties to the ScriptedReader.
I will try what you mention to access SchemaRegistry and send you some
comments.
On Wed, Apr 8, 2020 at 10:30 PM Matt Burgess wrote:
> As a follow up, while implementing [1] I ran into [2] which is a
> larger issue but affects the RecordSetWriters more than Readers as
> they add more properties that are expected to come (in our case) from
> the script. That makes things a bit more complicated, but I am still
> looking into it. If you are coding a ScriptedReader you might be ok
> with the guidance I gave earlier, but I have my doubts that trying to
> use a ScriptedRecordSetWriter with a SchemaRegistry will work right
> now. I'll post any findings in either/both [1] and [2], and as always
> I welcome all comments, questions, and suggestions :)
>
> Regards,
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-7343
> [2] https://issues.apache.org/jira/browse/NIFI-5115
>
> On Wed, Apr 8, 2020 at 8:19 PM Matt Burgess wrote:
> >
> > Jairo,
> > We should probably move this to the dev list since we're getting into
> > the NiFi API, but I wasn't sure you were subscribed, if you are let me
> > know and we can move it over there.
> >
> > StandardConfigurationContext is in framework-core so you should not
> > count on being able to use that class per se. Instead you should add a
> > property like the one in SchemaAccessUtils:
> >
> > public static final PropertyDescriptor SCHEMA_REGISTRY = new
> > PropertyDescriptor.Builder()
> > .name("schema-registry")
> > .displayName("Schema Registry")
> > .description("Specifies the Controller Service to use for the
> > Schema Registry")
> > .identifiesControllerService(SchemaRegistry.class)
> > .required(false)
> > .build();
> >
> > Then from your code you can add that property in your overridden
> > getSupportedPropertyDescriptors() method, and when you need it you can
> > get the property from the context, but don't refer to it as a
> > StandardConfigurationContext, there's an interface that has the same
> > methods and is in the API that you have access to already:
> >
> > final SchemaRegistry schemaRegistry =
> >
> context.getProperty(SCHEMA_REGISTRY).asControllerService(SchemaRegistry.class)
> >
> > Note that we're not getting the registry directly from the context,
> > rather we're getting the property value from the context and then
> > finding the corresponding SchemaRegistry, I believe that's done "under
> > the hood" by the ControllerServiceLookup, the
> > ".asControllerService(SchemaRegistry.class)" is the key part there.
> >
> > From there you can use the SchemaRegistry interface methods to get at
> > the other things, for example:
> >
> > descriptor = getPropertyDescriptor(SCHEMA_ACCESS_STRATEGY.getName());
> > schemaAccess = context.getProperty(descriptor).getValue();
> > schemaAccessStrategy = getSchemaAccessStrategy(schemaAccess,
> > schemaRegistry, context);
> >
> > Note that last method getSchemaAccessStrategy() is from
> > SchemaAccessUtils which you don't currently have access to, so you
> > might want to just copy all that over to your script. If you follow
> > the calls and code from SchemaRegistryService, you should be able to
> > bring any/all of that into your ScriptedReader so it can do the same
> > kind of processing.
> >
> > Regards,
> > Matt
> >
> > On Wed, Apr 8, 2020 at 7:03 PM Jairo Henao
> wrote:
> > >
> > > Thanks Matt,
> > >
> > > I have made some changes using AvroReader as an example, but I am
> still able to access other ControllerServices.
> > >
> > > From the RecordReaderFactory I have access to a context that is of
> type org.apache.nifi.controller.service.StandardConfigurationContext but
> with this I don't know how to access the AvroSchemaRegistry.
> > >
> > > Do you know how I can access the ControllerServiceLookup?
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Apr 8, 2020 at 3:19 PM Matt Burgess
> wrote:
> > >>
> > >> Jairo,
> > >>
> > >> It is possible for a ScriptedReader to use a SchemaRegistry, although
> > >> it isn't currently as easy as it could be. When the Scripted
> > >> Reader/Writer components were added [1], much of the reusable cod
Re: How to read a schema from a ScriptedReader
Thanks Matt, I have made some changes using AvroReader as an example, but I am still able to access other ControllerServices. >From the RecordReaderFactory I have access to a context that is of type *org.apache.nifi.controller.service.StandardConfigurationContext* but with this I don't know how to access the AvroSchemaRegistry. Do you know how I can access the ControllerServiceLookup? On Wed, Apr 8, 2020 at 3:19 PM Matt Burgess wrote: > Jairo, > > It is possible for a ScriptedReader to use a SchemaRegistry, although > it isn't currently as easy as it could be. When the Scripted > Reader/Writer components were added [1], much of the reusable code for > schema registry processing was in the service implementation NAR, so > the aforementioned Jira explains that using the SchemaRegistry > interface is certainly possible in ScriptedReader but fairly > cumbersome. Since then the SchemaRegistryService and associated util > classes were moved to a nifi-avro-record-utils [2] so > extensions/components can leverage schema registries. Unfortunately > nifi-avro-record-utils is not currently included in the scripting NAR, > but I have written up [3] to add this. > > In the meantime, take a look at SchemaRegistryService, > SchemaAccessUtils, and an implementation class such as AvroReader, you > could copy and paste all the necessary code to get your ScriptedReader > to interact with a schema registry. > > Regards, > Matt > > [1] https://issues.apache.org/jira/browse/NIFI-3734 > [2] https://issues.apache.org/jira/browse/NIFI-5123 > [3] https://issues.apache.org/jira/browse/NIFI-7343 > > On Wed, Apr 8, 2020 at 2:01 PM Jairo Henao > wrote: > > > > Hi all, > > > > Is there a way from a ScriptedReader (Controller Service) to read a > schema that is stored in an AvroSchemaRegistry? > > > > I am using Groovy as a language. > > > > Additionally, how can I read additional properties that I have added to > my ControllerService from Groovy code. > > > > Thanks in advance > > > > > > -- > > Saludos > > > > Jairo Henao > -- Saludos Jairo Henao *Chat Skype: jairo.henao.05*
How to read a schema from a ScriptedReader
Hi all, Is there a way from a ScriptedReader (Controller Service) to read a schema that is stored in an AvroSchemaRegistry? I am using Groovy as a language. Additionally, how can I read additional properties that I have added to my ControllerService from Groovy code. Thanks in advance -- Saludos Jairo Henao
Suggestions about the best way to count the status of several flows
Hi all, I have a flow that is split into several fragments, processed one by one and finally mixed to notify the status. If the MergeContent processor only maintains the common attributes, what would be the best way to count the final state of each flow (Successful, Failed, Ignored)? Maybe I use a distributed cache? But I only need one number for each state Nifi counters would need to be cleaned with ExecuteScript so that they do not accumulate data between runs. I have the option of accumulating in a database table, but I would like something lighter. Thanks -- Saludos
Re: Split JSON using an expression to define the PATH
Thanks Matt, What you say helps me understand even because EvaluateJsonPath doesn't support expressions either. I will try what you mention, I appreciate your ideas. On Tue, Feb 18, 2020 at 4:04 PM Matt Burgess wrote: > Jairo, > > IIRC the reason we don't support Expression Language (EL) for the > JSONPath expression is because the two DSLs use the same characters in > different syntax, such as $. To support both, I believe the user would > have to escape the JSONPath $ characters so the NiFi Expression > Language lexer doesn't think it's a NiFi expression. We could try what > some other languages do which is to catch those kinds of errors and > proceed under the assumption that the character must be part of a > JSONPath rather than an EL expression, but in an error condition how > would it know that it was a JSONPath error vs a NiFi EL error? I > believe the decision was made to keep things simple for the user and > so EL is not currently supported for that field. Happy to continue the > discussion though, and I welcome all opinions. > > In the meantime you may find (as Pierre suggested) that ForkRecord or > SplitRecord would work, as you can use a RecordPath expression rather > than a JSONPath expression. If the incoming JSON has a schema > (inferred or explicit) and is a top-level array of JSON objects (or > one JSON after another), then SplitRecord doesn't need a JSONPath as > the JsonTreeReader will take care of reading in the individual > records. If you need a JSONPath because you need to split on an array > "further down" in the input, you can try ForkRecord with a "Mode" set > to "Extract" and likely "Include Parent Fields" set to "false". > However as Simon mentioned, any transformation over the entire JSON > (whether it be to fork a nested array or JoltTransformJson) will > likely have to read the entire file into memory. > > Regards, > Matt > > > On Tue, Feb 18, 2020 at 3:32 PM Jairo Henao > wrote: > > > > Hi, > > Sorry, I should have given a little more details. > > > > My requirement is that I have a PG (Processor Group) that I defined as a > template, the PG contains several processors and one of them is a > SplitJSON, I want to receive the JsonPath to be applied in a flowfile > attribute. > > > > Apparently then, I must transform my JSON before entering the PG to > always find the same JsonPath. > > > > Do you consider it useful to make the change request in the processor so > that in the next versions it has that functionality? > > > > On Tue, Feb 18, 2020 at 2:02 PM Pierre Villard < > pierre.villard...@gmail.com> wrote: > >> > >> Hi, > >> > >> We would need a bit more details about what you try to achieve (an > example maybe?) but the record processors (SplitRecord, ForkRecord, etc) > might be useful. > >> > >> Thanks, > >> Pierre > >> > >> Le lun. 17 févr. 2020 à 23:55, Simon Bence > a écrit : > >>> > >>> Hi, > >>> > >>> Without knowing the actual use case: what if you would run the JSON > through JoltTransformJSON and convert it into a format would work well with > the SplitJson? It might help. (On the side note: as you mention “huge” > JSON, it might be a resource consuming operation) > >>> > >>> Regards, > >>> Bence > >>> > >>> On 2020. Feb 17., at 21:32, Jairo Henao > wrote: > >>> > >>> Hi all, > >>> > >>> Is there any way to be able to "split" a huge JSON, using as JsonPATH > the value of an attribute? > >>> > >>> SplitJson processor does not support this, it must be "hard-coded" > >>> > >>> > >>> -- > >>> Jairo Henao > >>> > >>> > > > > > > -- > > Jairo Henao > > > -- Saludos Jairo Henao *Chat Skype: jairo.henao.05*
Re: Split JSON using an expression to define the PATH
Hi, Sorry, I should have given a little more details. My requirement is that I have a PG (Processor Group) that I defined as a template, the PG contains several processors and one of them is a SplitJSON, I want to receive the JsonPath to be applied in a flowfile attribute. Apparently then, I must transform my JSON before entering the PG to always find the same JsonPath. Do you consider it useful to make the change request in the processor so that in the next versions it has that functionality? On Tue, Feb 18, 2020 at 2:02 PM Pierre Villard wrote: > Hi, > > We would need a bit more details about what you try to achieve (an example > maybe?) but the record processors (SplitRecord, ForkRecord, etc) might be > useful. > > Thanks, > Pierre > > Le lun. 17 févr. 2020 à 23:55, Simon Bence a > écrit : > >> Hi, >> >> Without knowing the actual use case: what if you would run the JSON >> through JoltTransformJSON and convert it into a format would work well with >> the SplitJson? It might help. (On the side note: as you mention “huge” >> JSON, it might be a resource consuming operation) >> >> Regards, >> Bence >> >> On 2020. Feb 17., at 21:32, Jairo Henao >> wrote: >> >> Hi all, >> >> Is there any way to be able to "split" a huge JSON, using as JsonPATH the >> value of an attribute? >> >> SplitJson processor does not support this, it must be "hard-coded" >> >> >> -- >> Jairo Henao >> >> >> -- Jairo Henao
Split JSON using an expression to define the PATH
Hi all, Is there any way to be able to "split" a huge JSON, using as JsonPATH the value of an attribute? SplitJson processor does not support this, it must be "hard-coded" -- Jairo Henao
