Re: Issue with AWS ELB on secure nifi-registry

The base file is here for comparison: https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23 On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende wrote: > For your first file, is what you showed there actually wrapped in >

Re: Issue with AWS ELB on secure nifi-registry

fi.cluster.node.address=nifi-dev.mobilgov.com > nifi.cluster.node.protocol.port= > nifi.cluster.node.protocol.threads=10 > nifi.cluster.node.event.history.size=25 > nifi.cluster.node.connection.timeout=5 sec > nifi.cluster.node.read.timeout=5 sec > nifi.cluster.firewall.f

Re: Issue with AWS ELB on secure nifi-registry

=US >> >> Is there an issue with using a wildcard cert? >> >> >>> On Mar 21, 2018, at 10:23 AM, Bryan Bende wrote: >>> >>> All identity strings are case & whitespace sensitive. >>> >>> The node identities in your authorizers.xml have n

Re: Issue with AWS ELB on secure nifi-registry

> > > I made the CN=*.{redacted}.com just like the self-signed certificate showed. > I now have a 1 node cluster up and running. It seems like NIfi isn’t taking > into account the wildcard and treating it as a “*” instead. > >> On Mar 21, 2018, at 11:33 AM, Bryan Bende wrot

Re: Issue with AWS ELB on secure nifi-registry

action="R"> > > > resource="/tenants" action="W"> > > > resource="/policies" action="R"> > > > resource="/policies" action="W"> >

Re: Issue with AWS ELB on secure nifi-registry

uot; action="R"> > > > resource="/policies" action="W"> > > > resource="/controller" action="R"> > > > resource="/cont

Re: GG kafka topic in avro format to NiFi

Hello, You don’t need to use a properties file. In your AvroRecordReader, just change the Schema Name property from ${schema.name} to the actual name of the schema you want to use from the schema registry. It just means that the record reader can only be used with one schema now, rather than tryi

Re: put pictures from remote server into hdfs

Hello, It would probably be best to use GetSFTP -> PutHDFS. No need to write the files out to local disk somewhere else with PutFile, they can go straight to HDFS. The filename in HDFS will be the "filename" attribute of the flow file, which GetSFTP should be setting to the filename it picked up

Re: Store Hash data type in redis

If you setup the RedisDistributedMapCacheClient, you can then use the PutDistributedMapCache processor to store anything you want in Redis. You could use the HashContent processor and then store that in Redis. -Bryan On Fri, Mar 23, 2018 at 9:03 AM, Mike Thomsen wrote: > I don't think there are

Re: HDFS Processors and Native Compression Codecs

Shawn, You are right, the issue is that the JVM only allows a native library to be loaded once, and furthermore if you have native methods that need to be linked to the native library, the native code must be in the same class loader where the native library is loaded (at least from my testing).

Re: Processor example with dynamically fetched values for attributs

Hello, This pattern isn't really supported at the moment... You would not want to make calls to external services when opening the config window of a processor because then you risk blocking on an network call if the system is down or unresponsive, or just risking taking a really long time to loa

Re: Nifi processor stops picking up file from queue

Hello, Approximately how many flow files are generated by the split? -Bryan On Mon, Mar 26, 2018 at 4:26 PM, af6140 wrote: > This is nifi 1.2.0. > > > > -- > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NIfi and Nifi Registry Connection Error

Just to double-check, when you defined the Registry Client in NiFi, did you enter the URL starting with "https" ? On Wed, Mar 28, 2018 at 3:13 PM, Scott Howell wrote: > I have a single node nifi server setup with a self-signed keystone and > truststore. I then used the same self-signed cert to c

Re: NIfi and Nifi Registry Connection Error

Ok, can you provide the stacktrace from nifi-app.log when it gets the socket timeout? On Wed, Mar 28, 2018 at 3:45 PM, Scott Howell wrote: > Yes I have https: defined on the registry client. > >> On Mar 28, 2018, at 2:29 PM, Bryan Bende wrote: >> >> Just to double-che

Re: NIfi and Nifi Registry Connection Error

er.java:141) > at > sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) > at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) > at > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java

Re: PutHDFS with mapr

If you want to see the classpath of a processor that has an "Additional Resources" property, you should be able to turn on TRACE logging for ExtensionManager in logback.xml: Wait 30 seconds, then modify the value of the "Additional Resources" property to force it to print. It will only print whe

Re: Processor example with dynamically fetched values for attributs

> > -- otherwise, the UI would have to call the NiFi server for refresh, from > time to time. > > Thanks. > > Regards, > Dominique > > > > 2018-03-26 17:59 GMT+02:00 Bryan Bende : >> >> Hello, >> >> This pattern isn't really supported at

Re: Getting Untrusted Proxy when logging into cluster

Standalone mode does not need the proxy permission so it is likely that the certificate of the standalone node also has the double-quotes, but it just doesn't matter. If you use keytool to list the contents of the keystore.jks, does it show the Owner with the double quotes? keytool --list -v -key

Re: Killing 'Stuck' Processors without restarting NiFi

Can you provide the portion of the thread dump for one of the stuck threads so we can see where it is getting stuck? This really shouldn't be normal occurrence. On Tue, Apr 3, 2018 at 12:10 AM, Pierre Villard wrote: > On my phone right now, can't provide much details, but there will be a way > t

Re: Cluster coordinator does not update cluster subdomain when new node states up

Did you rebuild all your ZooKeeper servers too, or just rebuilt NiFi servers point at the same existing ZooKeeper? On Thu, Apr 5, 2018 at 2:16 PM, Andy LoPresto wrote: > I suppose it could be a weird routing issue with your DNS settings or a > modified /etc/hosts file? Can you flush the DNS cache

Re: PUT/flow/process-groups/{id} API

I'm not sure if this is helpful to compare to what you are trying to do, but in the CLI that is soon to be released with 1.6.0, we have a similar command for "pg-start". Here is how it is implemented: https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-cli/src/main/java/org/apach

Re: Ranger-plugin authorises "anonymous" for /flow

Hello, I don't see any issue with the code you linked to. It's saying "if the ranger policies say the operation is allowed, then return approved". Is '{USER}' a special syntax in Ranger? or are you using that as a placeholder in email so you don't have to provide the real user identity? I haven'

Re: Ranger-plugin authorises "anonymous" for /flow

t; > which is used by certain matcher classes. > > On Mon, Apr 9, 2018 at 3:28 PM, Bryan Bende wrote: >> >> Hello, >> >> I don't see any issue with the code you linked to. It's saying "if the >> ranger policies say the operation is allowed, the

Re: Ranger-plugin authorises "anonymous" for /flow

[1] > https://community.hortonworks.com/questions/142667/how-to-give-permissions-to-users-to-access-nifi-ui.html > > On Mon, Apr 9, 2018 at 4:09 PM, Bryan Bende wrote: >> >> Ah thanks for the info, didn't know that. >> >> So you have a policy in Ranger whe

Re: Ranger-plugin authorises "anonymous" for /flow

thing else the authorizer would check if the incoming identity is "anonymous" and reject the request before even checking the actual policies from Ranger. On Mon, Apr 9, 2018 at 11:10 AM, Bryan Bende wrote: > Ok, so you end up in the NiFi UI with the identity in the top right > sayin

Re: Ranger-plugin authorises "anonymous" for /flow

quot;anonymous" and reject the request before even checking the actual >>policies from Ranger. > > Would this be a good setting to have in general, perhaps for managed > authorizer or in nifi.properties, so that it could be toggled for any > configured authorizer? >

Re: Referencing a property from nifi.properties into a processor.

You can also set system properties in bootstrap.conf, this was the primary way of setting up variables before either of the variable registries existed. java.arg.123=-Dmyvar=myvalue On Thu, Apr 19, 2018 at 8:41 AM, Matt Burgess wrote: > If you are using a version of NiFi that has ExecuteScript,

Re: Security issue when programmatically trying to create DBCP Controller Service

There should be an exception in nifi-app.log for the message you received: "Unable to create component to verify if it references any Controller Services" If you can find the stacktrace for that exception in nifi-app.log and provide it here, it may be helpful to see what is going on. On Fri, Ap

Re: Hadoop Processors Leak Classes

Is the behavior the same in 1.5.0, or only in 1.2.0? We can't undo the instance class loading for Hadoop processors unless we undo the static usage of UserGroupInformation. I think it is possible to go back to the non-static usage since we the kerberos issues were actually resolved by the useSubj

Re: Hadoop Processors Leak Classes

>> loading (if possible) would fix the problem. If instance class loading is >> needed, then finding the reason that classloader can't be cleaned up would >> be the fix. >> >> >> On Fri, Apr 27, 2018 at 8:38 AM Bryan Bende wrote: >>> >>>

Re: Efficiently caching API results in a NiFi controller service

Tim, The reason the DMC works the way it does is because the cached data needs to be shared across a cluster. For example, a processor like DetectDuplicate needs to detect duplicates across all NiFi nodes and not just the local node, or the same thing with Wait/Notify. In your case I don't think

Re: Efficiently caching API results in a NiFi controller service

I would say that yes you should plan for concurrent access... Since you have multiple processors that will be utilizing the same service that means multiple threads, and even if you have one processors calling it then if you increased the concurrent tasks of that processor then multiple threads wi

Re: Am I doing this right? with regarding to records

Unfortunately the current JSON record readers are not expecting a JSON document per line because technically that is not a valid JSON document itself. Your file would have to be represented as an array of documents like [ doc1, doc2, doc3, ...] There is a PR up to support the per-line JSON documen

Re: Am I doing this right? with regarding to records

MergeRecord using correlation attribute: STATE > > PartitionRecord also adds an attribute with the value it partitioned so I'll > get an attribute STATE="FL" for example and after I Merge them I can ingest > in to something like /data/$STATE/ > > I think I have

Re: Am I doing this right? with regarding to records

Isn't that what PartitionRecord + RouteOnAttribute already does? On Tue, May 1, 2018 at 4:58 PM, Otto Fowler wrote: > Maybe a group and dispatch processor would help, > > JsonPath -> key > Group by key > key -> route > > > On May 1, 2018 at 16:30:28, Bryan Be

Re: Can a NiFi processor depend on multiple independent controller services?

It is possible if you create a chain of service API NARs... For example, API NAR 2 depends on API NAR 1 depends on standard services API NAR. A processor NAR that depends on APi NAR 2 will then see API 2, API 1, and all standard services API. On Wed, May 2, 2018 at 11:51 PM, Tim Dean wrote: >

Re: Fetch Contents of HDFS Directory as a Part of a Larger Flow

Another option would be if the Hadoop client was installed on the NiFi node then you could use one of the script processors to make a call to "hadoop fs -ls ...". If the response is so large that it requires heavy lifting of writing out temp tables to HDFS and then fetching those files into NiFi,

Re: Fetch Contents of HDFS Directory as a Part of a Larger Flow

oesn't support inputs either. > > > Thanks > > Shawn > > > From: Bryan Bende > Sent: Thursday, May 3, 2018 10:51:03 AM > To: users@nifi.apache.org > Subject: Re: Fetch Contents of HDFS Directory as a Part of a Larger Flow > > Anoth

Re: Fetch Contents of HDFS Directory as a Part of a Larger Flow

I think the point of the original thread was that building a REST service in NiFi wasn't necessarily meant to be used as the back-end of a full-blown web application that is used by hundreds/thousands of users. However there are still plenty of situations where it is certainly fine, especially for

Re: Getting error in Kafka Consumer

Hello, What is the Schema Access Strategy set to in your AvroRecordSetWriter? Since you said you have an embedded schema, I would expect your reader's Schema Access Strategy to be set to "Embedded" and then the writer would be set to "Inherit from Reader" or "Embedded". Maybe you have something d

Re: Error Reference When Creating Template From PG Under NiFi Registry Version Control

The 1.6 toolkit should work fine with 1.5.0, even the latest CLI from master should work with 1.5 or 1.6. Even though I do agree that we should be moving away from using templates for deployment, I do think we should be clearing the version control DTO when creating a template, it was not left in

Re: Error Reference When Creating Template From PG Under NiFi Registry Version Control

FYI, created a JIRA and PR here: https://issues.apache.org/jira/browse/NIFI-5163 On Mon, May 7, 2018 at 10:03 AM, Bryan Bende wrote: > The 1.6 toolkit should work fine with 1.5.0, even the latest CLI from > master should work with 1.5 or 1.6. > > Even though I do agree that we shou

Re: Improving Ranger audit logging in Nifi

Hello, If you are talking about what is being audited to Solr, then that can't be changed without changing the code. Thanks, Bryan On Tue, May 8, 2018 at 2:27 AM, wrote: > Hello Joe, > > Good to hear that the memory leak was fixed in the 1.6 series. > > One more question remains: Is there any

Re: Faster way to develop/test custom NiFi processors?

Boris, There is a very extensive mock test framework for processors, the docs are here [1]. Also tons of examples of unit testing processors in the source code for the standard processors [2]. You should be able to have very high confidence in the processor logic using this approach. Hope that

Re: Allowing all users to connect

You could create an identity mapping in nifi.properties that maps any cert DN to a hard coded user identity, say “nifi-user”, so every user looks like that one user. You would lose ability to track which users changed the flow, but I’m guessing that’s not important if you want to give everyone acc

Re: Allowing all users to connect

It really comes down to how you want to configure the authorizer... An authorizer is made up of a user-group-provider and a policy-provider. The user-group-provider can be file-based, ldap, composite (combination of multiple), or custom if you implement your own. The policy provider can be file-

Re: Allowing all users to connect

for LDAP and > Kerberos." > > > On Thu, May 17, 2018 at 10:21 AM Bryan Bende wrote: >> >> It really comes down to how you want to configure the authorizer... >> >> An authorizer is made up of a user-group-provider and a policy-provider. >> >>

Re: NiFi bootstrap loop issue

Have not seen that... The bootstrap process will restart the main NiFi process, but only if the main process successfully started up and got a PID and then went down. What part are you seeing looping? is it possible the main process keeps going down and getting restarted? On Mon, May 21, 2018 at

Re: NiFi bootstrap loop issue

can even log into the admin console briefly when it > reaches that point. > > On Mon, May 21, 2018 at 2:35 PM Bryan Bende wrote: >> >> Have not seen that... >> >> The bootstrap process will restart the main NiFi process, but only if >> the main process suc

Re: InvokeScriptedProcessor from a shared folder

Is there a stacktrace in nifi-app.log at the time you got the validation error? On Tue, May 22, 2018 at 4:35 PM, Boris Tyukin wrote: > I tried both along with nifi restart. > > it kinda works now. But this is CRAZY. If someone can explain this, I would > give a hug :) > > My steps: > > 1. Drop ne

Re: Zero Downtime Deployment

Hello, The short answer is that you currently can't do a true zero downtime deployment of NARs. The longer answer... When a node joins the cluster there is a comparison done between the node's flow and the node's NARs against the cluster's flow and cluster's NARs. In the case of the flow, if the

Re: NiFi 1.6.0 fails to start

Hello, There is some work being done to support Java 9/10, but at this time NiFi currently requires Java 8. There is an issue when using Java 9/10 where some javax.xml classes are now part of the JDK, but before NiFi had to bundle them, so it creates conflicting classes. Thanks, Bryan > On

Re: NiFi Web UI in split-horizon DNS environments

Hi Matt, What version of NiFi is this happening with? -Bryan On Fri, Jun 8, 2018 at 1:03 PM, Matthew McDowell wrote: > Hi everyone, > > > > I’m using NiFi in an environment where the server running the NiFi UI is > reached via different hostnames depending on the where the user is coming > from

Re: TCP Connection Limit Error

Hello, There is a counter that is incremented when a connection is opened, and decremented when it is closed. When this counter exceeds the number configured in the processor, then it rejects the connection with the message you are seeing. Since you have the max connections set to 1000, there must

Re: TCP Connection Limit Error

uggest upgrade the version may solve this issue? > > > > Thanks and Regards, > > *Rajesh Biswas* | +91 9886433461 | www.bridgera.com > > > > *From:* Bryan Bende [mailto:bbe...@gmail.com] > *Sent:* Friday, June 15, 2018 12:50 AM > *To:* users@nifi.apache.org

Re: Consumekafka error I

I’m not sure if this will help, but you mentioned you are using Kafka (1.0.0.3.1.0). NiFi has processors that correspond to the Kafka version, and generally it is best to use the client version that matches the broker version. So you should be using ConsumeKafka_1_0 which uses the Kafka 1.0.0 c

Re: FetchParquet Type Conversion Error

Joe, There was a similar issue I worked on recently that had to do with array handling: https://issues.apache.org/jira/browse/NIFI-5316 From a quick glance this may be a different variation of a similar issue. We are using Parquet’s Java API th

Re: Registry - deleting versions of flow

I think we need to implement what you described in option #2 which is basically like a force push in git terminology. If you want to create a JIRA for that I think it is a good request. > On Jun 29, 2018, at 6:40 AM, Fredrik Skolmli wrote: > > Hi everyone. > > I'm not sure if my workflow need

Re: NiFi ExecuteScript vs multiple processors vs custom processor

Boris, Regarding templates being limited... templates were really made as a way to share example flows, or help with debugging if you need to send someone your flow. Unfortunately they turned into a deployment mechanism since there wasn't a better solution at the time. Using NiFi Registry should

Re: NiFi ExecuteScript vs multiple processors vs custom processor

ith registry 0.1 it was > not possible to version nested PGs within parent PGs so I could not have > "templatized" PG which has it is own version and use that PG with other > versioned PGs. Has it changed with registry 0.2 now out? > > On Tue, Jul 10, 2018 at 11:08 AM Bry

Re: Separating custom service API from its implementation

Tim, In the case where it is your own custom service api and service impl, and you know you are never going to have another implementation of the API, then it doesn't really matter and having them all in one NAR will work. The issue is that by bundling the implementation with the API, now someone

Re: AVRO is the only output format with ExecuteSQL

I would also add that the pattern of splitting to 1 record per flow file was common before the record processors existed, and generally this can/should be avoided now in favor of processing/manipulating records in place, and keeping them together in large batches. On Tue, Aug 7, 2018 at 9:10 AM,

Re: configuration variables?

The variable registry is hierarchical, so variables defined in a parent group will be visible to all children, unless a child group has a variable with the same name which will override the parent's value. So you can put any common variables in the root group. On Tue, Aug 7, 2018 at 9:40 AM, l vi

Re: Multiple registry instances sharing Git repo

Mike, It is not really made to have multiple instances pointing at the same git repo. Each registry has a metadata DB and a flow storage component which can be filesystem or git. So the metadata DB won't know about the stuff created in the other instance. Generally it is either a single shared

Re: [EXT] Re: Get all Processors

You could probably achieve the same thing but traversing the process groups and asking each one for its processors without the includeDescendantGroups=true. It would be more complex on the client side, but would avoid making one huge request. On Tue, Aug 14, 2018 at 11:07 AM, Karthik Kothareddy (

Re: Simple CSV to Parquet without Hadoop

Scott, Unfortunately the Parquet API itself is tied to the Hadoop Filesystem object which is why NiFi can't read and write Parquet directly to flow files (i.e. they don't provide a way to read/write to/from Java input and output streams). The best you can do is trick the Hadoop API into using the

Re: Simple CSV to Parquet without Hadoop

ion: No FileSystem for scheme > > BTW, I'm using NiFi version 1.5 > > Thanks, > Scott > > > On Tue, Aug 14, 2018 at 12:44 PM, Bryan Bende wrote: >> >> Scott, >> >> Unfortunately the Parquet API itself is tied to the Hadoop Filesystem >> o

Re: Simple CSV to Parquet without Hadoop

get the >>> > following errors: >>> > >>> > Failed to locate the winutils binary in the hadoop binary path >>> > IOException: Could not locate executable null\bin\winutils.exe in the >>> > Hadoop >>> > binar

Re: ConvertAvroSchema with unmapped fields

Have you tried using ConvertRecord instead of ConvertAvroSchema? ConvertAvroSchema comes from the Kite bundle which has a few other processors like ConvertCsvToAvro and ConvertJsonToAvro, all of which existed before NiFi's record processing capabilities. In general the record processors should be

Re: Managing Secrets in NiFi Registry

Hello, You are correct that currently variables cannot be marked as sensitive, and for that reason they currently shouldn't be used to store sensitive values since they will be stored in plain-text in the flow.xml.gz and also in the versioned flows saved to registry. I do think the concept of sen

Re: Nifi Publish/Consumer Kafka and Azure Event Hub

Hello, Any user defined properties in the processor should be passed along to Kafka so you should be able to add a new property with the name sasl.mechanism and the value PLAIN, without that the processors will assume GSSAPI. On Wed, Sep 5, 2018 at 9:55 PM João Henrique Freitas wrote: > > Hi Ja

Re: Invoke HTTP custom headers.

Hello, The "Attributes to Send" property is a regular expression, so you should be able to put ".*" to send all flow file attributes as headers, or you can make a pattern that matches a group of attributes that follow some naming convention. Thanks, Bryan On Mon, Sep 10, 2018 at 6:22 AM, Dnyane

Re: Nifi Registry + Git: target registry ignores git flows

Assuming you have a dev DB & git repo and then a prod DB & git repo, each DB goes with the corresponding repo and has ids that are unique to the environment, so you can't take one of the DBs and point at the other’s git repo, that is why you need the export/import process with the CLI. The reas

Re: unsubscribe

To unsubscribe from the users list you must send an email to users-unsubscr...@nifi.apache.org [1] . [1] https://nifi.apache.org/mailing_lists.html On Tue, Sep 11, 2018 at 10:45 PM, Laurie McIntosh wrote: > unsubscribe > > On Wed, Sep 12, 2018 at 3:58 AM Mark W. Webb wrote: >> >> unsubscribe

Re: question about ListFile processor

Hello, Yes the way you described it is correct. If the directory is local then the state is local because no other node could take over since the files are only on that node. When the directory is remote then it is intended to be run on primary node and the state is stored cluster wide so that i

Re: Running Nifi in cluster mode

Hello, There is a graph of processors often referred to as the "flow" and each node in the cluster runs a copy of the flow. So all nodes are running the same components, with the exception of source processors that happen to be scheduled as primary node only. The data must be divided across the c

Re: Nifi Registry + Git: target registry ignores git flows

:133) > [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317] > > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) > [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317] > > at > org.eclipse.jetty.util.thread.QueuedThr

Re: Certificate Issue when connecting to NiFi Registry

In addition to what Mark said, the NiFi nodes to be represented as users somewhere in order to create policies that grant them proxy permission. If your authorizer is using the LDAP user group provider then you could create users in LDAP to represent your NiFi nodes. If you don't want to create u

Re: Mergecontent on Cluster

Hello, I'm not sure I follow... wouldn't it be more efficient to merge multiple files in parallel across the cluster? If you had to converge them all to one node, then this doesn't seem much different than just having a stand-alone NiFi, which would go against needing a cluster to achieve the des

Re: Certificate Issue when connecting to NiFi Registry

is how it’s intended to work? > > Thank You. > > Best Regards, > Nikhil C. > > On 17/9/18 20:39, "Bryan Bende" wrote: > > In addition to what Mark said, the NiFi nodes to be represented as > users somewhere in order to create policies that grant t

***UNCHECKED*** Re: Case insensitive NiFi Login

Hello, I don't think you can make it truly case insensitive, but you can use the identity mappings to transform the incoming identity to either upper case or lower case: nifi.security.identity.mapping.transform.dn=LOWER Assuming you make nifi.security.identity.mapping.pattern.dn match your LDAP

Re: upload template with REST problem

The URL you specified is not the correct path, it is missing "process-groups". http://10.10.100.105:8080/nifi-api/process-groups/f2ae3d20-e105-37a1--/templates/upload On Wed, Sep 19, 2018 at 4:17 PM l vic wrote: > > Hi, > I have template for the processing group "test-template.xml

Re: Whiltelable Apache NiFi UI

I think this would require making your own custom build of NiFi with your changes applied to the UI. The only thing I know of that you can do without changing code is to specify "banner text" in nifi.properties which will display across the very top of the UI. nifi.ui.banner.text= On Wed, Sep 19

Re: upload template with REST problem

That means there is no process group with the id you specified. On Thu, Sep 20, 2018 at 12:36 PM l vic wrote: > > yes, in this case i see another error: > > Unable to locate group with id > '46f95fe5-3024-31d8--'.ma6219-mac:deploy > > > On Thu, S

Re: upload template with REST problem

ot;id" must actually be "groupId", then it works... As usual, Nifi > documentation is vague or misleading > > On Thu, Sep 20, 2018 at 12:38 PM Bryan Bende wrote: >> >> That means there is no process group with the id you specified. >> On Thu, Sep 20, 2018

Re: upload template with REST problem

ttached, if i am missing > more verbose part I apologize > > On Thu, Sep 20, 2018 at 12:53 PM Bryan Bende wrote: >> >> The REST API documentation [1] for the method you are using says: >> >> NameLocation Type Description >> id path

Re: How to Use ListHDFS File Filter Regex

Shawn, I believe this issue [1] is the fix you are looking for, but unfortunately isn't in a release yet, but should be in the next one. As a work around, you may be able to list everything with ListHDFS and then send the results to a RouteOnContent that routes any content matching your .bat rege

Re: A sensible approach to scheduling via the API?

The main issue is that although NiFi has a lot of similar functionality to a workflow ETL tool, it is really a data flow tool that is meant to be used in more of a streaming fashion where there is no concept of "finished" or "complete" because there is a constant flow of data. >From a scheduling p

Re: ***UNCHECKED*** Re: Case insensitive NiFi Login

horizations.xml file and restarted nifi to check if login works. > > > > Regards, > Dnyaneshwar Pawar > > > -Original Message- > From: Bryan Bende > Sent: Wednesday, September 19, 2018 6:40 PM > To: users@nifi.apache.org > Subject: ***UNCHECKED***

Re: Listing S3

Hi Martijn, The request for the "list" processors to support incoming flow files comes up frequently. The issue is that the list processors are meant to continuously watch a given directory/bucket and maintain state about what has been seen and only find newer stuff. So if you let the processor su

Re: Listing S3

ut doing something similar for List processors? If there is no >> > incoming connection, then they continue to behave as they always have. >> > If there is an incoming connection and no flow file, no work is >> > performed. If there is an incoming connection with available

Re: Listing S3

ff the state? Or maintain the > state as is but just don't consider it? > > - > Sivaprasanna > > On Tue, Sep 25, 2018 at 7:54 PM Bryan Bende wrote: >> >> I like that approach too. >> On Tue, Sep 25, 2018 at 10:21 AM Pierre Villard >> wrote: >> >

Re: Nifi-registry and upgrade Nifi environments

Scott, I would think that you would upgrade dev to the latest version, in this case 1.7.1, then make sure all your flows are working properly and commit any updates to the registry. For example any of the process groups that involved PutSFTP would need to be committed because of the new properties

Re: Dynamic Mapping

How are you currently defining schemas and transforms? or are you free to choose depending what works best for building the data flow? If you are willing to use JOLT to define the transform then you have a few options... 1) As Joe mentioned, you can send the GUID in a header which becomes a flow

Re: Maximum Memory for NiFi?

Generally the larger the heap, the more likely to have long GC pauses. I'm surprised that you would need a 70GB heap given NiFi's design where the content of the flow files is generally not held in memory, unless many of the processors you are using are not written in an optimal way to process the

Re: variable url for remote process group

The RPG URL can be edited after importing to the next environment, and it should be ignored when comparing against the flow in registry so it won’t be seen as a change that needs to be committed. So although you can’t use a variable, you should still be able to version control a flow with an RPG

Re: putsolrcontentstream and kerberos

I've only done it against Solr cloud, but I don't know a reason why it wouldn't work against standalone Solr. Nothing is jumping out at me as being wrong with your config. My JAAS config was the following: SolrJClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true ke

Re: Issue with RouteOnAttribute

Hello, You cannot reference flow file content from expression language, so whatever you want to route on will need to be extracted into a flow file attribute. You can use EvaluateJsonPath with something like status = $.result.status then in RouteOnAttribute ${status:equals('OK')} Thanks, Bryan

Re: NiFi fails on cluster nodes

The cluster configuration section of the admin guide [1] is independent of whether it is embedded or external zookeeper. The only real difference is you won't set nifi.state.management.embedded.zookeeper.start=true, but besides that you all of the other config would be the same whether using embed

Re: NiFi fails on cluster nodes

This is not related to ZooKeeper... I think you are missing something related to TLS/SSL configuration, maybe you set cluster protocol to be secure, but then you didn't configure NiFi with a keystore/truststore? On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen wrote: > > Not sure what's going on here

<    1   2   3   4   5   6   7   8   9   >