RE: [EXTERNAL] - Re: FW: Loading nifi server within a Java process

[Jamie Wang]

Thank James for the pointers. I will try and let everyone know what I find.

Jamie

-Original Message-
From: James Wing [mailto:jvw...@gmail.com] 
Sent: Wednesday, April 05, 2017 3:55 PM
To: NiFi Dev List
Subject: [EXTERNAL] - Re: FW: Loading nifi server within a Java process

Jamie,

Sorry you didn't get a response.  I'm happy to confirm that this is the correct 
list for your question.  There is no well-beaten path for running NiFi in 
another process, NiFi assumes that it is deployed as it's own process.  NiFi 
contains a number of features for classloader isolation, thread management, and 
performance monitoring that may mean something different in another process, 
although I can't say for sure they will break.

Hopefully, some others can chime in on the pitfalls.

Thanks,

James

On Wed, Apr 5, 2017 at 2:25 PM, Jamie Wang  wrote:

> Hi,
>
> I sent the message below to the user group and unfortunately I didn't 
> get any response. I am forwarding the message to this list and am 
> hoping to get some feedback. If this is the wrong list for this 
> message, I apologize and please let me know the right place to post. Thank 
> you in advance.
>
> Jamie
>
> From: Jamie Wang [mailto:jam...@opentext.com]
> Sent: Tuesday, April 04, 2017 12:53 PM
> To: us...@nifi.apache.org
> Subject: [EXTERNAL] - Loading nifi server within a Java process
>
> I have a situation where we have constraint on the number of processes 
> we can use and hence, is it possible to load nifi server within 
> another Java process? Looking at the 
> org.apache.nifi.bootstrap.RunNiFi.java, it seems that I can create a 
> Java object that does most of what
> org.apache.nifi.bootstrap.RunNiFi.main() do. Any thoughts if this will 
> work? What are some of the things I need to watch out? And is there 
> another better way to do this or an example? Thank you for your time.
>
> Jamie
>

Re: Ensuring org.slf4j.helpers.MessageFormatter.arrayFormat() in

[Russell Bateman]

Here's a full example of what I see when I run a unit test. In my root 
/pom.xml/, I'm forcing a dependency of /slf4j-api/ and /slf4-simple/, 
version 1.7.25, to ensure what's there isn't something old offered by 
another dependency. Any suggestion would be welcome.



*java.lang.AssertionError: java.lang.NoSuchMethodError: 
org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;*


at 
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:193)
at 
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:152)
at 
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:147)
at 
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:142)
at 
com.imatsolutions.nifi.processor.SlaTimerTest.testStopTimestampWarnAndPass(SlaTimerTest.java:114)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)

at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
*Caused by: java.lang.NoSuchMethodError: 
org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;*

at org.apache.nifi.util.CapturingLogger.warn(CapturingLogger.java:304)
at org.apache.nifi.util.CapturingLogger.warn(CapturingLogger.java:299)
at org.apache.nifi.util.CapturingLogger.warn(CapturingLogger.java:294)
at 
org.apache.nifi.util.MockComponentLog.warn(MockComponentLog.java:137)
at 
com.imatsolutions.nifi.processor.SlaTimer.onTrigger(SlaTimer.java:109)
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at 
org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:243)
at 
org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:237)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)





On 04/06/2017 12:28 PM, Russell Bateman wrote:
I'm struggling to figure out how to respond to this problem of 
ensuring the presence of MessageFormatter(), which I think is in 
/slf4j-simple-x.y.z.jar/, and/or ensuring the log4j system is properly 
initialized. The need for slf4j 

Re: org.apache.nifi.processor.util.FlowFileSessionWrapper deprecated

[Joe Witt]

Russell,

Sorry there was no reply but to help ensure it doesn't look like this
tree feel in an empty forest thanks for following up.  You're right
that this refactoring occurred.  There was some powerful session
merging capability made available that allowed us to simplify how
processors like MergeContent worked under the covers.

Thanks
Joe

On Thu, Apr 6, 2017 at 4:53 PM, Russell Bateman  wrote:
> In answer to my own question, ... we discovered that the Binclass has been
> refactored to obviate the FlowFileSessionWrapper. Some light refactoring of
> our consuming code was required, that's all.
>
> Thanks.
>
>
> On 04/03/2017 02:22 PM, Russell Bateman wrote:
>>
>> I've tried building using NiFi JARs from 1.1.1 and 1.1.0, but only 1.0.0
>> works. It appears, unless I'm mistaken, that /MergeConte//n//t/ requires it
>> and the code appears to be in 1.1.2. (Maybe it's just Monday.) Here's the
>> error I'm getting back from Maven:
>>
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile
>> (default-compile) on project standard-processors: Compilation failure:
>> Compilation failure:
>> [ERROR]
>> /home/russ/sandboxes/nifi-pipeline.v73_release.dev/code/nifi-pipeline/standard-processors/src/main/java/com/imatsolutions/nifi/processor/SearchServerBatch.java:[34,38]
>> cannot find symbol
>> [ERROR] symbol:   class FlowFileSessionWrapper
>> [ERROR] location: package org.apache.nifi.processor.util
>>
>> Thanks.
>>
>> On 04/03/2017 02:05 PM, Russell Bateman wrote:
>>>
>>> We're moving our code base ahead from NiFi 1.0.0 to 1.1.2 and have some
>>> classes that are using FlowFileSessionWrapper. Appears to wrap up a flowfile
>>> and its corresponding session; our use was in some special batching code. I
>>> could roll my own, I guess, but there may be a very good reason not to,
>>> i.e.: a better way to handle what it did for us, stay out of trouble, etc.
>>>
>>> So far, Googling encourages me to believe that this has been deprecated
>>> and I shouldn't use it, but so far I haven't stumbled upon why or what to do
>>> differently. Can anyone throw some light on this?
>>>
>>> Thanks.
>>
>>
>

Re: org.apache.nifi.processor.util.FlowFileSessionWrapper deprecated

[Russell Bateman]

In answer to my own question, ... we discovered that the Binclass has 
been refactored to obviate the FlowFileSessionWrapper. Some light 
refactoring of our consuming code was required, that's all.


Thanks.

On 04/03/2017 02:22 PM, Russell Bateman wrote:
I've tried building using NiFi JARs from 1.1.1 and 1.1.0, but only 
1.0.0 works. It appears, unless I'm mistaken, that /MergeConte//n//t/ 
requires it and the code appears to be in 1.1.2. (Maybe it's just 
Monday.) Here's the error I'm getting back from Maven:


[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile 
(default-compile) on project standard-processors: Compilation failure: 
Compilation failure:
[ERROR] 
/home/russ/sandboxes/nifi-pipeline.v73_release.dev/code/nifi-pipeline/standard-processors/src/main/java/com/imatsolutions/nifi/processor/SearchServerBatch.java:[34,38] 
cannot find symbol

[ERROR] symbol:   class FlowFileSessionWrapper
[ERROR] location: package org.apache.nifi.processor.util

Thanks.

On 04/03/2017 02:05 PM, Russell Bateman wrote:
We're moving our code base ahead from NiFi 1.0.0 to 1.1.2 and have 
some classes that are using FlowFileSessionWrapper. Appears to wrap 
up a flowfile and its corresponding session; our use was in some 
special batching code. I could roll my own, I guess, but there may be 
a very good reason not to, i.e.: a better way to handle what it did 
for us, stay out of trouble, etc.


So far, Googling encourages me to believe that this has been 
deprecated and I shouldn't use it, but so far I haven't stumbled upon 
why or what to do differently. Can anyone throw some light on this?


Thanks.

Ensuring org.slf4j.helpers.MessageFormatter.arrayFormat() in

[Russell Bateman]

I'm struggling to figure out how to respond to this problem of ensuring 
the presence of MessageFormatter(), which I think is in 
/slf4j-simple-x.y.z.jar/, and/or ensuring the log4j system is properly 
initialized. The need for slf4j is clearly coming from NiFi; I only need 
it because it's a dependency in NiFi. I know there are problems with 
slf4j version mixtures and I'm trying not to do anything else but allow 
NiFi to cause the right version to get pulled in.


When I run some of my unit tests, I'm seeing this:


log4j:WARN No appenders could be found for logger 
(com.imatsolutions.nifi.processor.filter.LegacyFullDocSearchReplaceFilter).

log4j:WARN Please initialize the log4j system properly.

java.lang.AssertionError: java.lang.NoSuchMethodError: 
org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;


at 
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:192)

.
.
.
Caused by: java.lang.NoSuchMethodError: 
org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;

at org.apache.nifi.util.CapturingLogger.warn(CapturingLogger.java:304)
.
.
.

Suggestions? Rules of thumb?

Thanks.

Re: [ANNOUNCE] New Apache NiFi Committer Bin Qiu

[Scott Aslan]

Congrats Bin!

On Wed, Apr 5, 2017 at 5:57 PM, Michael Moser  wrote:

> Welcome aboard, Bin, and congratulations.
>
>
> On Wed, Apr 5, 2017 at 11:44 AM, Marc  wrote:
>
> > Congrats and well deserved for everything you've done!
> >
> > On Wed, Apr 5, 2017 at 11:42 AM, Bryan Rosander 
> > wrote:
> >
> > > Congrats Bin!
> > >
> > > On Wed, Apr 5, 2017 at 9:52 AM, Oleg Zhurakousky 
> > wrote:
> > >
> > > > And of course I did it in the wrong thread ;)
> > > > Congrats Bin!!!
> > > >
> > > > > On Apr 5, 2017, at 9:48 AM, Oleg Zhurakousky 
> > wrote:
> > > > >
> > > > > Thank you all!
> > > > >
> > > > >  > > > >> *Joe Percivall*
> > > > >> e: joeperciv...@gmail.com
> > > > >
> > > >
> > > >
> > >
> >
>



-- 
*Scott Aslan = new WebDeveloper(*
*{"location": {"city": "Saint Cloud","state": "FL",
"zip": "34771"},"contact": {"email":
"scottyas...@gmail.com ","linkedin":
"http://www.linkedin.com/in/scottyaslan
"}});*

Re: [ANNOUNCE] New Apache NiFi PMC Member - Oleg Zhurakousky

[Scott Aslan]

Congrats Oleg!!!

On Wed, Apr 5, 2017 at 5:56 PM, Michael Moser  wrote:

> This is great news, congrats Oleg!
>
>
>
> On Wed, Apr 5, 2017 at 11:41 AM, Bryan Rosander 
> wrote:
>
> > Congrats Oleg!
> >
> > On Wed, Apr 5, 2017 at 9:53 AM, Oleg Zhurakousky <
> ozhurakou...@pivotal.io>
> > wrote:
> >
> > > Thank you all!
> > >
> > >  > > > *Joe Percivall*
> > > > e: joeperciv...@gmail.com
> > >
> > >
> >
>



-- 
*Scott Aslan = new WebDeveloper(*
*{"location": {"city": "Saint Cloud","state": "FL",
"zip": "34771"},"contact": {"email":
"scottyas...@gmail.com ","linkedin":
"http://www.linkedin.com/in/scottyaslan
"}});*

Re: Policy and UI

[Matthew Clarke]

Mark,

The Global Policy "access users/user groups" has both "view" and "modify"
options.  In order for a user to be able to "access all policies", they
must at least have view to "access users/user groups".  This is why
"Policies" remains greyed out for your user.

Matt

On Thu, Apr 6, 2017 at 7:56 AM, Mark Bean  wrote:

> We just setup a new cluster. A user has been granted:
>
> Global policy to access all policies (read and write)
> Global policy to access the controller (read)
> Global policy to view system diagnostics (read)
> Global policy to view the user interface (read)
> Admin policy for process group NiFi Flow (read)
> Component policy for process group NiFi Flow (read)
>
> Despite those policies, the Global Menu > Policies option is still grayed
> out and inaccessible.
>
> I don't think this is the same issue, but we discovered a browser-specific
> problem related to policies and functionality as well. using Chrome 36.x,
> (a different user) could select Global Menu > Policies (it wasn't grayed
> out), but nothing happened. When he switched to FireFox 31.6.0, it worked
> as expected; the Access Polices page came up when selecting it from the
> menu.
>
> So, the first problem appears to be related to proper policy/permission
> settings. I'm not understanding what policy is missing to allow the viewing
> of Polices. The second is a browser-specific issue. I'm wondering if this
> is a known issue.
>
> Please advise.
>
> Thanks,
> Mark
>

Policy and UI

[Mark Bean]

We just setup a new cluster. A user has been granted:

Global policy to access all policies (read and write)
Global policy to access the controller (read)
Global policy to view system diagnostics (read)
Global policy to view the user interface (read)
Admin policy for process group NiFi Flow (read)
Component policy for process group NiFi Flow (read)

Despite those policies, the Global Menu > Policies option is still grayed
out and inaccessible.

I don't think this is the same issue, but we discovered a browser-specific
problem related to policies and functionality as well. using Chrome 36.x,
(a different user) could select Global Menu > Policies (it wasn't grayed
out), but nothing happened. When he switched to FireFox 31.6.0, it worked
as expected; the Access Polices page came up when selecting it from the
menu.

So, the first problem appears to be related to proper policy/permission
settings. I'm not understanding what policy is missing to allow the viewing
of Polices. The second is a browser-specific issue. I'm wondering if this
is a known issue.

Please advise.

Thanks,
Mark

Re: Installing Nifi server

[Joe Witt]

Hello

The ui is only ever a client side construct.  Just like you plan to
programmatically interact with the REST API your web browser does the same
on the users behalf as they view the app and click around.

You should be all set.

Thanks
Joe

On Apr 6, 2017 7:21 AM, "Vdsa"  wrote:

I need to install only NIFi server, I do not require nifi UI. I will be
handling nifi server with rest API. Is it possible to install only NIFI
server without UI?



--
View this message in context: http://apache-nifi-developer-
list.39713.n7.nabble.com/Installing-Nifi-server-tp15398.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Installing Nifi server

[Vdsa]

I need to install only NIFi server, I do not require nifi UI. I will be
handling nifi server with rest API. Is it possible to install only NIFI
server without UI?



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Installing-Nifi-server-tp15398.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.