Processor classpath

2017-07-04 Thread James Srinivasan
Hi, I'm developing a processor which needs to read some of its config from the classpath. Reading the docs etc., NiFi's classpath is a little funky - where's the best (least worst?) location for such files? I note that the HDFS processors can read their config (core-site.xml etc) from the classpat

Re: Processor classpath

2017-07-05 Thread James Srinivasan
x27;t bundled in the actual classpath at all. If that is the functionality > you want I suggest following the template from the HDFS processors as you > stated. Otherwise just carryon as usual and pretend its a regular java > application. > > On Tue, Jul 4, 2017 at 4:31 PM, James Sriniv

Re: Processor classpath

2017-07-07 Thread James Srinivasan
t; Configuration config = new Configuration(); > config.addResource(new Path(resource.trim())); > > Hope that helps. > > -Bryan > > > > On Wed, Jul 5, 2017 at 4:15 PM, James Srinivasan > wrote: >> Thanks, I ended up stracing the NiFi process to see where i

Re: Processor classpath

2017-07-07 Thread James Srinivasan
s, > > I'm saying that it should not find a core-site.xml in NiFi's conf > directory because the conf directory is not on the classpath of > processors. > > Are you saying you have tested putting it there and believe it is finding it? > > -Bryan > > On

Re: Processor classpath

2017-07-07 Thread James Srinivasan
spath of the system class loader should be what you see in > nifi-bootstrap.log for the command that started NiFi. > > Sorry for the confusion. > > > On Fri, Jul 7, 2017 at 12:03 PM, James Srinivasan > wrote: >> Here's where strace tells me my processor is looking for

Re: Getting untrusted proxy message while trying to setup secure NIFI cluster

2017-07-13 Thread James Srinivasan
Hi, I found I had to add this to authorizations.xml for R & W, with corresponding users.xml entries: Still not entirely sure my secured cluster is fully set up correctly - planning on writing up how we did it tho. James On 13 July 2017

Processor using Kerberos keytab auth - can't renew TGT

2017-07-14 Thread James Srinivasan
Hi all, I have a NiFi processor which uses Kerberos keytab authentication to write data to Accumulo. I have a separate thread which periodically runs in order to try renewing my TGT (UserGroupInformation.getCurrentUser.checkTGTAndReloginFromKeytab()). This code works fine outside NiFi, but inside

Re: Processor using Kerberos keytab auth - can't renew TGT

2017-07-14 Thread James Srinivasan
ds, > Georg > > Joe Witt schrieb am Fr., 14. Juli 2017 um 15:12 Uhr: >> >> James, >> >> I know Jeff Storck has recently been doing some work around >> Kerberos/TGT renewal. Hopefully he can share some of his >> observations/work back on this thread soo

Re: Processor using Kerberos keytab auth - can't renew TGT

2017-07-14 Thread James Srinivasan
running services like HBase as well. > > James Srinivasan schrieb am Fr., 14. Juli 2017 > um 15:36 Uhr: >> >> Hi Georg, >> >> I am indeed using open-jdk8 on CentOS 7.3, but I'm not sure why my >> standalone app is ok, whereas the same code in NiFi isn

Re: GetHTTP 403:Forbidden

2017-10-11 Thread James Srinivasan
Hi, Doesn't sound like you have the certs set up correctly since --insecure for curl skips certificate validation. I'm not aware of a similar option for NiFi, but assuming you generated the certificate yourself, searching for something like "java https self signed web" should help. If the certific

Re: GetHTTP 403:Forbidden

2017-10-12 Thread James Srinivasan
is with curl and > --insecure. Does anyone know of a way in NIFI to do the equivalence of > "--insecure" ? The curl that works is: > curl --noproxy "*" --insecure --cacert ./ca.pem --cert ./c_cert.pem --key > ./c_key.pem https://xxx.mitre.org/xx/xx > > thank you &

Incorrect PublishKafka_0_10 documentation?

2017-11-07 Thread James Srinivasan
I've been struggling to get NiFi working with Kerberos authenticated Kafka. According to the docs, the "Kerberos Service Name" property specifies: "The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config. Corresponds to Kafka's 'securi

NiFi cluster with DistributedMapCacheServer/Client

2018-04-13 Thread James Srinivasan
Hi all, Is there a recommended way to set up a DistributedMapCacheServer/Client on a cluster, ideally with some amount of HA (NiFi 1.3.0)? I'm using a shared persistence directory, and when adding and enabling the controller it seems to start on my primary node (but not the other two - status keep

Re: NiFi cluster with DistributedMapCacheServer/Client

2018-04-13 Thread James Srinivasan
there now too that might be interesting (not in 1.3.0 > perhaps but..). > > We offered a simple out of the box one early and to ensure the > interfaces are right. Since then the community has popped up some > real/stronger implementations like you're mentioning. > > Thanks

Re: Fetch Contents of HDFS Directory as a Part of a Larger Flow

2018-05-03 Thread James Srinivasan
We handle a similar situation using CTAS and then retrieve the resulting data using webhdfs. James On Thu, 3 May 2018, 17:18 Bryan Bende, wrote: > The two step idea makes sense... > > If you did want to go with the OS call you would probably want > ExecuteStreamCommand. > > On Thu, May 3, 2018

NiFi ExecuteScript vs multiple processors vs custom processor

2018-07-09 Thread James Srinivasan
Hi all, I was wondering if there is any general guidance about when to use ExecuteScript and when to use a chain of processors? For example, in one application I am downloading a HTML index file, extracting the links corresponding to more index pages of data per year, fetching those pages, extract

NiFi protobuf processor

2018-08-16 Thread James Srinivasan
I've written a quick NiFi protobuf to json processor. Before I go through my organisation's admin to contribute it, is it likely to be accepted? Mostly because I'm lazy, it doesn't use the new record functionality and only supports protobuf decoding. Thanks James

Re: NiFi protobuf processor

2018-08-20 Thread James Srinivasan
hanks > On Thu, Aug 16, 2018 at 2:17 PM Otto Fowler wrote: > > > > You should send this to the dev list > > > > > > On August 16, 2018 at 13:22:29, James Srinivasan > > (james.sriniva...@gmail.com) wrote: > > > > I've written a quick NiFi protobu

Re: Nifi Publish/Consumer Kafka and Azure Event Hub

2018-09-05 Thread James Srinivasan
I've not tried this myself, but once you have a working JAAS config (from https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs#send-and-receive-messages-with-kafka-in-event-hubs), set the corresponding protocol and mechanism properties in the NiFi proces

Demuxing NiFi logs

2018-09-07 Thread James Srinivasan
As we add more and more different processor groups doing separate things, our nifi-app log is getting rather unmanageable. I'm probably missing something obvious, but is one log per group possible? Thanks

Re: Demuxing NiFi logs

2018-09-07 Thread James Srinivasan
really meant to be the true historical > record of value. The data provenance entries are. Can you share more > about what you're trying to accomplish? > > Thanks > On Fri, Sep 7, 2018 at 2:47 PM James Srinivasan > wrote: > > > > As we add more and more differen

Re: Demuxing NiFi logs

2018-09-07 Thread James Srinivasan
rience. > > But your point/ask makes sense. > On Fri, Sep 7, 2018 at 3:04 PM James Srinivasan > wrote: > > > > Sure, we have several data flow managers who are each responsible for a > number of separate data flows, each of which lives in a separate processor > group. Wh

Generating Remote URL for InvokeHTTP

2018-11-16 Thread James Srinivasan
Hi all, I'm observing some slightly unusual behaviour with my flow and wanted to run a possible explanation past the list. I'm using NiFi to scrape a website consisting of nested data e.g. GET http://server/2018/16/11/ returns a webpage full of links to today's data I'm using a combination of I

Re: stop processing related flowfiles

2018-11-28 Thread James Srinivasan
Hopefully you already know this: "Kafka only provides a total order over records *within* a partition, not between different partitions in a topic. Per-partition ordering combined with the ability to partition data by key is sufficient for most applications. However, if you require a total order o

NiFi JSON enrichment

2018-12-17 Thread James Srinivasan
Hi all, I'm trying to enrich a data stream using NiFi. So far I have the following: 1) Stream of vehicle data in JSON format containing (id, make, model) 2) This vehicle data goes into HBase, using id as the row key and the json data as the cell value (cf:json) 3) Stream of position data in JSON

Re: NiFi JSON enrichment

2018-12-18 Thread James Srinivasan
asy as in no > work at all, but given your limitations with the NiFi version could be done > sooner maybe. > > Andrew > > On Mon, Dec 17, 2018, 9:53 AM James Srinivasan > wrote: >> >> Hi all, >> >> I'm trying to enrich a data stream using NiFi. S

Re: NiFi JSON enrichment

2018-12-18 Thread James Srinivasan
> > James, > > > > The easiest would be to merge json in a custom processor. Not easy as in no > > work at all, but given your limitations with the NiFi version could be done > > sooner maybe. > > > > Andrew > > > > On Mon, Dec 17, 2018

Re: NiFi JSON enrichment

2019-01-03 Thread James Srinivasan
on demand >> >> >> On December 18, 2018 at 15:04:30, Otto Fowler (ottobackwa...@gmail.com) >> wrote: >> >> You could implement a custom schema registry that converts the protos to >> schema on the fly and caches. >> >> >> On December 18,

Re: expression failure in URL concatenation?

2019-01-31 Thread James Srinivasan
Out of interest, does the URL containing the double slash work? On Thu, 31 Jan 2019, 21:42 l vic I am using processor group variable as base part of my URL: > REST_URL=http://localhost:8080/nifi-api > > I am trying to append second part of URL in InvokeHTTP regardless if > REST_URL ends with '/',

Re: Is the DistributedMapCacheService a single point of failure?

2019-02-12 Thread James Srinivasan
We switched to HBase_1_1_2_ClientMapCacheService for precisely this reason. It works great (we already had HBase which probably helped) On Tue, 12 Feb 2019 at 12:51, Vos, Walter wrote: > > Hi, > > I'm on NiFi 1.5 and we're currently having an issue with one of the nodes in > our three node clust

Re: Is the DistributedMapCacheService a single point of failure?

2019-02-12 Thread James Srinivasan
ver runs on all nodes, but it doesn't replicate data between them, > and the client can only point at one of these nodes. If you have to > switch the client to point at a new server, then the cache will be > starting over on the new server. > > On Tue, Feb 12, 2019 at 8:11 AM

Re: Tailor logback.xml custom Appender and Logging

2019-02-25 Thread James Srinivasan
I suggested something similar here: http://apache-nifi-users-list.2361937.n4.nabble.com/Demuxing-NiFi-logs-td5689.html On Mon, 25 Feb 2019 at 12:11, James McMahon wrote: > > A very helpful approach to redirecting logging for a particular processor > type is presented here: > https://community.

Re: Tailor logback.xml custom Appender and Logging

2019-02-25 Thread James Srinivasan
I created a JIRA to log this here: https://issues.apache.org/jira/browse/NIFI-6079 Please do comment - not sure my suggested approach is best On Mon, 25 Feb 2019 at 20:19, Andy LoPresto wrote: > > Hate to be “that guy”, but the simplest thing I can think of right now is set > up a NiFi flow th

Re: [EXTERNAL] Re: PublishKafka_1_0 Kerberos

2019-02-26 Thread James Srinivasan
If someone is updating the Kafka docs, it would be great to get this corrected: http://apache-nifi-users-list.2361937.n4.nabble.com/Incorrect-PublishKafka-0-10-documentation-td3406.html (Been on my to-do list for ages) On Tue, 26 Feb 2019, 16:16 Bryan Bende, wrote: > Hi Dan, > > There is a not

Re: Different NiFi Node sizes within same cluster

2019-03-06 Thread James Srinivasan
Yes, we hit this with the new load balanced queues (which, to be fair, we also had with remote process groups previously). Two "old" nodes got saturated and their queues filled while three "new" nodes were fine. My "solution" was to move everything to new hardware which we had inbound anyway. On

Re: Different NiFi Node sizes within same cluster

2019-03-06 Thread James Srinivasan
t; workload for slower nodes. > > Thanks > > On Wed, Mar 6, 2019 at 3:48 PM James Srinivasan > wrote: >> >> Yes, we hit this with the new load balanced queues (which, to be fair, we >> also had with remote process groups previously). Two "old" nodes

Re: Different NiFi Node sizes within same cluster

2019-03-06 Thread James Srinivasan
> constrained on CPU. Easier said than done unfortunately. > > > > On Wed, Mar 6, 2019 at 4:23 PM James Srinivasan > > wrote: > >> > >> In our case, backpressure applied all the way up to the TCP network > >> source which meant we lost data. AIUI, current

Re: Connecting to a kerberized HBase 2 instance

2019-03-15 Thread James Srinivasan
That combo works for me. AIUI, the data API is compatible but the management API (table creation etc) isn't. On Fri, 15 Mar 2019, 17:29 Mike Thomsen, wrote: > Can the 1.1.2 client connect to a kerberized HBase 2 instance? We're stuck > on NiFi 1.8.0 for now and someone upgraded us to HBase 2.0 f

Re: Connecting to a kerberized HBase 2 instance

2019-03-15 Thread James Srinivasan
On Fri, Mar 15, 2019 at 1:44 PM Bryan Bende wrote: >> >> Yes, James's statement is what I believe to be true as well. >> >> On Fri, Mar 15, 2019 at 1:33 PM James Srinivasan >> wrote: >> > >> > That combo works for me. AIUI, the data API is compa

Re: ListenUDP: internal queue at maximum capacity, could not queue event

2019-06-05 Thread James Srinivasan
Presumably you'd want to mirror the stream to all nodes for when the primary node changes? On Wed, 5 Jun 2019, 13:46 Bryan Bende, wrote: > The processor is started on all nodes, but onTrigger method is only > executed on the primary node. > > This is something we've discussed trying to improve b

Re: ListenUDP: internal queue at maximum capacity, could not queue event

2019-06-05 Thread James Srinivasan
handle that? > > On Wed, Jun 5, 2019 at 11:30 AM James Srinivasan > wrote: > > > > Presumably you'd want to mirror the stream to all nodes for when the > primary node changes? > > > > On Wed, 5 Jun 2019, 13:46 Bryan Bende, wrote: > >> > >&g

Kerberos Ticket Renewal (when not updating Hadoop user)

2019-06-12 Thread James Srinivasan
Hi all, I'm finally getting around to fixing up some deprecation issues with our use of Kerberos with Accumulo and GeoMesa (https://github.com/locationtech/geomesa/). Because I didn't know any better at the time, I used the KerberosToken ctor specifying that the Hadoop user should be replaced. Com

Re: Kerberos Ticket Renewal (when not updating Hadoop user)

2019-06-13 Thread James Srinivasan
we explicitly try to do a relogin. NIFI-5134 [3] > contains more information on that. > > Hope this information helps! > > [1] https://github.com/apache/nifi/pull/2360 > [2] https://issues.apache.org/jira/browse/NIFI-3472 > [3] https://issues.apache.org/jira/browse/N

Re: Kerberos Ticket Renewal (when not updating Hadoop user)

2019-06-13 Thread James Srinivasan
ion on that. >>> >>> Hope this information helps! >>> >>> [1] https://github.com/apache/nifi/pull/2360 >>> [2] https://issues.apache.org/jira/browse/NIFI-3472 >>> [3] https://issues.apache.org/jira/browse/NIFI-5134 >>> >>> On W

Re: Custom Processor Upgrade

2019-08-15 Thread James Srinivasan
I find strace (or procmon for Windows) very handy to debug such resource loading issues. On Thu, 15 Aug 2019, 19:02 Bryan Bende, wrote: > I was making sure you didn't have any code that was dependent on the > internal structure of how the NARs are unpacked. > > I can't explain why it can't find

Re: Regarding setting up multiple DistribitedMapCacheServer controler service

2020-12-16 Thread James Srinivasan
If you are running on a cluster, you might want to consider an alternative such as HBase_2_ClientMapCacheService otherwise the node running the DistributedMapCacheServer becomes a SPOF. (others: please correct me if I am wrong, this was the case a while back when I moved to using the HBase server

Re: Problem with the GetFile processor deleting my entire installation

2021-06-04 Thread James Srinivasan
You can use the two permissions detailed here https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.GetFile/index.html On Fri, 4 Jun 2021, 21:11 Ruth, Thomas, wrote: > No matter how secure I make the cluster, what’s to pre

Re: Error on nifi start

2022-12-13 Thread James Srinivasan
I think nifi currently supports java 8 or 11, not 17: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_requirements On Tue, 13 Dec 2022, 12:45 James McMahon, wrote: > I am using an Ansible role from Ansible GALAXY that has been tested and > validated up through Apach

Re: UI SocketTimeoutException - heavy IO

2023-03-22 Thread James Srinivasan
Apologies in advance if I've got this completely wrong, but I recall that error if I forget to increase the limit of open files for a heavily loaded install. It is more obvious via the UI but the logs will have error messages about too many open files. On Wed, 22 Mar 2023, 16:49 Mark Payne, wrote

Re: Configuring ExecuteStreamCommand on jar flowfiles

2023-12-03 Thread James Srinivasan
Since a jar file is mostly just a standard zip file, can you use a built in processor instead? On Sun, 3 Dec 2023, 15:36 James McMahon, wrote: > I have a large volume of a wide variety of incoming data files. A subset > of these are jar files. Can the ExecuteStreamCommand be configured to run >