Hi,
I'm developing a processor which needs to read some of its config from
the classpath. Reading the docs etc., NiFi's classpath is a little
funky - where's the best (least worst?) location for such files? I
note that the HDFS processors can read their config (core-site.xml
etc) from the classpat
x27;t bundled in the actual classpath at all. If that is the functionality
> you want I suggest following the template from the HDFS processors as you
> stated. Otherwise just carryon as usual and pretend its a regular java
> application.
>
> On Tue, Jul 4, 2017 at 4:31 PM, James Sriniv
t; Configuration config = new Configuration();
> config.addResource(new Path(resource.trim()));
>
> Hope that helps.
>
> -Bryan
>
>
>
> On Wed, Jul 5, 2017 at 4:15 PM, James Srinivasan
> wrote:
>> Thanks, I ended up stracing the NiFi process to see where i
s,
>
> I'm saying that it should not find a core-site.xml in NiFi's conf
> directory because the conf directory is not on the classpath of
> processors.
>
> Are you saying you have tested putting it there and believe it is finding it?
>
> -Bryan
>
> On
spath of the system class loader should be what you see in
> nifi-bootstrap.log for the command that started NiFi.
>
> Sorry for the confusion.
>
>
> On Fri, Jul 7, 2017 at 12:03 PM, James Srinivasan
> wrote:
>> Here's where strace tells me my processor is looking for
Hi,
I found I had to add this to authorizations.xml for R & W, with
corresponding users.xml entries:
Still not entirely sure my secured cluster is fully set up correctly -
planning on writing up how we did it tho.
James
On 13 July 2017
Hi all,
I have a NiFi processor which uses Kerberos keytab authentication to
write data to Accumulo. I have a separate thread which periodically
runs in order to try renewing my TGT
(UserGroupInformation.getCurrentUser.checkTGTAndReloginFromKeytab()).
This code works fine outside NiFi, but inside
ds,
> Georg
>
> Joe Witt schrieb am Fr., 14. Juli 2017 um 15:12 Uhr:
>>
>> James,
>>
>> I know Jeff Storck has recently been doing some work around
>> Kerberos/TGT renewal. Hopefully he can share some of his
>> observations/work back on this thread soo
running services like HBase as well.
>
> James Srinivasan schrieb am Fr., 14. Juli 2017
> um 15:36 Uhr:
>>
>> Hi Georg,
>>
>> I am indeed using open-jdk8 on CentOS 7.3, but I'm not sure why my
>> standalone app is ok, whereas the same code in NiFi isn
Hi,
Doesn't sound like you have the certs set up correctly since
--insecure for curl skips certificate validation. I'm not aware of a
similar option for NiFi, but assuming you generated the certificate
yourself, searching for something like "java https self signed web"
should help. If the certific
is with curl and
> --insecure. Does anyone know of a way in NIFI to do the equivalence of
> "--insecure" ? The curl that works is:
> curl --noproxy "*" --insecure --cacert ./ca.pem --cert ./c_cert.pem --key
> ./c_key.pem https://xxx.mitre.org/xx/xx
>
> thank you
&
I've been struggling to get NiFi working with Kerberos authenticated
Kafka. According to the docs, the "Kerberos Service Name" property
specifies:
"The Kerberos principal name that Kafka runs as. This can be defined
either in Kafka's JAAS config or in Kafka's config. Corresponds to
Kafka's 'securi
Hi all,
Is there a recommended way to set up a
DistributedMapCacheServer/Client on a cluster, ideally with some
amount of HA (NiFi 1.3.0)? I'm using a shared persistence directory,
and when adding and enabling the controller it seems to start on my
primary node (but not the other two - status keep
there now too that might be interesting (not in 1.3.0
> perhaps but..).
>
> We offered a simple out of the box one early and to ensure the
> interfaces are right. Since then the community has popped up some
> real/stronger implementations like you're mentioning.
>
> Thanks
We handle a similar situation using CTAS and then retrieve the resulting
data using webhdfs.
James
On Thu, 3 May 2018, 17:18 Bryan Bende, wrote:
> The two step idea makes sense...
>
> If you did want to go with the OS call you would probably want
> ExecuteStreamCommand.
>
> On Thu, May 3, 2018
Hi all,
I was wondering if there is any general guidance about when to use
ExecuteScript and when to use a chain of processors? For example, in
one application I am downloading a HTML index file, extracting the
links corresponding to more index pages of data per year, fetching
those pages, extract
I've written a quick NiFi protobuf to json processor. Before I go through
my organisation's admin to contribute it, is it likely to be accepted?
Mostly because I'm lazy, it doesn't use the new record functionality and
only supports protobuf decoding.
Thanks
James
hanks
> On Thu, Aug 16, 2018 at 2:17 PM Otto Fowler wrote:
> >
> > You should send this to the dev list
> >
> >
> > On August 16, 2018 at 13:22:29, James Srinivasan
> > (james.sriniva...@gmail.com) wrote:
> >
> > I've written a quick NiFi protobu
I've not tried this myself, but once you have a working JAAS config
(from
https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs#send-and-receive-messages-with-kafka-in-event-hubs),
set the corresponding protocol and mechanism properties in the NiFi
proces
As we add more and more different processor groups doing separate things,
our nifi-app log is getting rather unmanageable. I'm probably missing
something obvious, but is one log per group possible?
Thanks
really meant to be the true historical
> record of value. The data provenance entries are. Can you share more
> about what you're trying to accomplish?
>
> Thanks
> On Fri, Sep 7, 2018 at 2:47 PM James Srinivasan
> wrote:
> >
> > As we add more and more differen
rience.
>
> But your point/ask makes sense.
> On Fri, Sep 7, 2018 at 3:04 PM James Srinivasan
> wrote:
> >
> > Sure, we have several data flow managers who are each responsible for a
> number of separate data flows, each of which lives in a separate processor
> group. Wh
Hi all,
I'm observing some slightly unusual behaviour with my flow and wanted
to run a possible explanation past the list. I'm using NiFi to scrape
a website consisting of nested data
e.g. GET http://server/2018/16/11/ returns a webpage full of links to
today's data
I'm using a combination of I
Hopefully you already know this:
"Kafka only provides a total order over records *within* a partition, not
between different partitions in a topic. Per-partition ordering combined
with the ability to partition data by key is sufficient for most
applications. However, if you require a total order o
Hi all,
I'm trying to enrich a data stream using NiFi. So far I have the following:
1) Stream of vehicle data in JSON format containing (id, make, model)
2) This vehicle data goes into HBase, using id as the row key and the
json data as the cell value (cf:json)
3) Stream of position data in JSON
asy as in no
> work at all, but given your limitations with the NiFi version could be done
> sooner maybe.
>
> Andrew
>
> On Mon, Dec 17, 2018, 9:53 AM James Srinivasan
> wrote:
>>
>> Hi all,
>>
>> I'm trying to enrich a data stream using NiFi. S
> > James,
> >
> > The easiest would be to merge json in a custom processor. Not easy as in no
> > work at all, but given your limitations with the NiFi version could be done
> > sooner maybe.
> >
> > Andrew
> >
> > On Mon, Dec 17, 2018
on demand
>>
>>
>> On December 18, 2018 at 15:04:30, Otto Fowler (ottobackwa...@gmail.com)
>> wrote:
>>
>> You could implement a custom schema registry that converts the protos to
>> schema on the fly and caches.
>>
>>
>> On December 18,
Out of interest, does the URL containing the double slash work?
On Thu, 31 Jan 2019, 21:42 l vic I am using processor group variable as base part of my URL:
> REST_URL=http://localhost:8080/nifi-api
>
> I am trying to append second part of URL in InvokeHTTP regardless if
> REST_URL ends with '/',
We switched to HBase_1_1_2_ClientMapCacheService for precisely this
reason. It works great (we already had HBase which probably helped)
On Tue, 12 Feb 2019 at 12:51, Vos, Walter wrote:
>
> Hi,
>
> I'm on NiFi 1.5 and we're currently having an issue with one of the nodes in
> our three node clust
ver runs on all nodes, but it doesn't replicate data between them,
> and the client can only point at one of these nodes. If you have to
> switch the client to point at a new server, then the cache will be
> starting over on the new server.
>
> On Tue, Feb 12, 2019 at 8:11 AM
I suggested something similar here:
http://apache-nifi-users-list.2361937.n4.nabble.com/Demuxing-NiFi-logs-td5689.html
On Mon, 25 Feb 2019 at 12:11, James McMahon wrote:
>
> A very helpful approach to redirecting logging for a particular processor
> type is presented here:
> https://community.
I created a JIRA to log this here:
https://issues.apache.org/jira/browse/NIFI-6079
Please do comment - not sure my suggested approach is best
On Mon, 25 Feb 2019 at 20:19, Andy LoPresto wrote:
>
> Hate to be “that guy”, but the simplest thing I can think of right now is set
> up a NiFi flow th
If someone is updating the Kafka docs, it would be great to get this
corrected:
http://apache-nifi-users-list.2361937.n4.nabble.com/Incorrect-PublishKafka-0-10-documentation-td3406.html
(Been on my to-do list for ages)
On Tue, 26 Feb 2019, 16:16 Bryan Bende, wrote:
> Hi Dan,
>
> There is a not
Yes, we hit this with the new load balanced queues (which, to be fair, we
also had with remote process groups previously). Two "old" nodes got
saturated and their queues filled while three "new" nodes were fine.
My "solution" was to move everything to new hardware which we had inbound
anyway.
On
t; workload for slower nodes.
>
> Thanks
>
> On Wed, Mar 6, 2019 at 3:48 PM James Srinivasan
> wrote:
>>
>> Yes, we hit this with the new load balanced queues (which, to be fair, we
>> also had with remote process groups previously). Two "old" nodes
> constrained on CPU. Easier said than done unfortunately.
> >
> > On Wed, Mar 6, 2019 at 4:23 PM James Srinivasan
> > wrote:
> >>
> >> In our case, backpressure applied all the way up to the TCP network
> >> source which meant we lost data. AIUI, current
That combo works for me. AIUI, the data API is compatible but the
management API (table creation etc) isn't.
On Fri, 15 Mar 2019, 17:29 Mike Thomsen, wrote:
> Can the 1.1.2 client connect to a kerberized HBase 2 instance? We're stuck
> on NiFi 1.8.0 for now and someone upgraded us to HBase 2.0 f
On Fri, Mar 15, 2019 at 1:44 PM Bryan Bende wrote:
>>
>> Yes, James's statement is what I believe to be true as well.
>>
>> On Fri, Mar 15, 2019 at 1:33 PM James Srinivasan
>> wrote:
>> >
>> > That combo works for me. AIUI, the data API is compa
Presumably you'd want to mirror the stream to all nodes for when the
primary node changes?
On Wed, 5 Jun 2019, 13:46 Bryan Bende, wrote:
> The processor is started on all nodes, but onTrigger method is only
> executed on the primary node.
>
> This is something we've discussed trying to improve b
handle that?
>
> On Wed, Jun 5, 2019 at 11:30 AM James Srinivasan
> wrote:
> >
> > Presumably you'd want to mirror the stream to all nodes for when the
> primary node changes?
> >
> > On Wed, 5 Jun 2019, 13:46 Bryan Bende, wrote:
> >>
> >&g
Hi all,
I'm finally getting around to fixing up some deprecation issues with
our use of Kerberos with Accumulo and GeoMesa
(https://github.com/locationtech/geomesa/). Because I didn't know any
better at the time, I used the KerberosToken ctor specifying that the
Hadoop user should be replaced. Com
we explicitly try to do a relogin. NIFI-5134 [3]
> contains more information on that.
>
> Hope this information helps!
>
> [1] https://github.com/apache/nifi/pull/2360
> [2] https://issues.apache.org/jira/browse/NIFI-3472
> [3] https://issues.apache.org/jira/browse/N
ion on that.
>>>
>>> Hope this information helps!
>>>
>>> [1] https://github.com/apache/nifi/pull/2360
>>> [2] https://issues.apache.org/jira/browse/NIFI-3472
>>> [3] https://issues.apache.org/jira/browse/NIFI-5134
>>>
>>> On W
I find strace (or procmon for Windows) very handy to debug such resource
loading issues.
On Thu, 15 Aug 2019, 19:02 Bryan Bende, wrote:
> I was making sure you didn't have any code that was dependent on the
> internal structure of how the NARs are unpacked.
>
> I can't explain why it can't find
If you are running on a cluster, you might want to consider an
alternative such as HBase_2_ClientMapCacheService otherwise the node
running the DistributedMapCacheServer becomes a SPOF.
(others: please correct me if I am wrong, this was the case a while
back when I moved to using the HBase server
You can use the two permissions detailed here
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.GetFile/index.html
On Fri, 4 Jun 2021, 21:11 Ruth, Thomas, wrote:
> No matter how secure I make the cluster, what’s to pre
I think nifi currently supports java 8 or 11, not 17:
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_requirements
On Tue, 13 Dec 2022, 12:45 James McMahon, wrote:
> I am using an Ansible role from Ansible GALAXY that has been tested and
> validated up through Apach
Apologies in advance if I've got this completely wrong, but I recall that
error if I forget to increase the limit of open files for a heavily loaded
install. It is more obvious via the UI but the logs will have error
messages about too many open files.
On Wed, 22 Mar 2023, 16:49 Mark Payne, wrote
Since a jar file is mostly just a standard zip file, can you use a built in
processor instead?
On Sun, 3 Dec 2023, 15:36 James McMahon, wrote:
> I have a large volume of a wide variety of incoming data files. A subset
> of these are jar files. Can the ExecuteStreamCommand be configured to run
>
50 matches
Mail list logo
