Re: Nifi ExecuteScript slow performance

[Vyshali]

Hi Matt,

I'm using Jython in executescript because of my requirement.I cant switch to
groovy because I'm using packages supported by Python.Is there any way to
increase the speed of the executescript processor.Please help me with your
ideas.

Thanks,
Vyshali



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Facing issue in Site to Site Https Communication

[Nishant Gupta]

Hi Team,

I am trying to make site to site communication between 2 standalone nifi
instances.
I am getting below error

2017-11-12 23:44:41,304 WARN [Remote Process Group
b1265abf-015f-1000-632e-e7939fbad38f: https://X.X.X.X:9443/nifi Thread-1]
o.a.n.remote.StandardRemoteProcessGroup Unable to connect to
RemoteProcessGroup[https://X.X.X.X:9443/nifi] due to
javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path validation failed:
java.security.cert.CertPathValidatorException: Path does not chain with any
of the trust anchors.

I have generated the certificate using below command and imported in
browser.:

sudo bash ./tls-toolkit.sh standalone -n 'server-1,server-2' -C 'CN=demo,
OU=nifi' -O -o ../security_output

I am not able to figure out any solution on google.

Can you please help me out.

Thanks and Regards,
Nishant Gupta

Re: Kubernetes

[Robert R. Bruno]

I'll be sure to share my experiences as well.  I found using Rancher
greatly aided in setting up a Kubernetes cluster on bare metal if that is
the route you are going.

On Sun, Nov 12, 2017, 12:27 john duggan  wrote:

> +1.
>
> I haven't deployed NiFi into a Kubernetes cluster, but am planning some
> time over the next week to setup a POC (Proof of Concept) environment. I'm
> certainly interested if anybody else already has experience in this area.
> Hopefully, I can share my experience over the next week or two.
>
>
> Regards,
> JpD.
>
> On 12 November 2017 at 16:44, Robert R. Bruno  wrote:
>
> > I wanted to check with the group to see if anyone has deployed NiFi to a
> > Kubernetes cluster.  If so, perhaps we could sync up on your lessons
> > learned.  I am about to start looking at this myself.  I am leaning
> towards
> > creating a NiFi Kubernetes Chart.  Figured worth checking here first
> before
> > starting.
> >
> > Thanks,
> > Robert
> >
>

Compiling MiNiFi C++

[Marc]

Hello,
  If you are compiling MiNIFI C++ to run on another platform and/or cross
compiling for another platform please use the cmake portable flag.

  This is traditionally done via cmake -DPORTABLE=ON ..

  I will be submitting a PR to add this to our readme. I'll also be adding
this to our "Customizing MiNIFI C++ wiki entry" [1] .

  The reason for this is that one of our dependencies uses the native
architecture flag ( -march=native). If you are transferring the binary to
another architecture, it may or may not have the microcode to support those
native ops. I will be adding a section to our wiki that covers cross
compilation a little more so this is obvious.


  Thanks,
   Marc

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74685143

Kubernetes

[Robert R. Bruno]

I wanted to check with the group to see if anyone has deployed NiFi to a
Kubernetes cluster.  If so, perhaps we could sync up on your lessons
learned.  I am about to start looking at this myself.  I am leaning towards
creating a NiFi Kubernetes Chart.  Figured worth checking here first before
starting.

Thanks,
Robert

Re: minifi-cpp: Expression language core API

[Marc]

Hi Andy,

   Sorry for the delayed response.

   I like the idea of moving the capabilities out of core so that we can
decouple components for a variety of reasons that boil down to minimizing
work to break apart components in the future. I'm certainly not against
augmenting core to facilitate functions like EL, but in my opinion, we
should be able to do so without directly coupling the core with the concept
of expression language. My hope would be that an injected component ( this
may necessitate a more complex framework like google fruit ) can help
inject an expression language specific lambda. I think this mostly goes
inline with your idea that we have a way to plug in an arbitrary evaluation
mechanism, but instead, I'm saying we should inject a function ( in a
controlled way, through a custom configuration ) to execute the evaluation.

Adding a function argument to getProperty that includes the argument for
FlowFile indicates that the processor developer wants to get the property
in relation to the FlowFile. This shouldn't require a separate type of
property but rather a lambda that was provided by the extension to evaluate
the property in relation to the FlowFile's attributes/and or content. If
the expression language is disabled, replaced by another, or the property
is not an expression this does nothing other return the property value.  In
my mind you would never have the FlowFile in onSchedule and if you pass it
with your getProperty, the developer naturally wants any evaluation to
occur if that property is an evaluation ). If they don't want a property to
ever evaluate, they needn't pass the FlowFile pointer.

  What you mention for cons vary wildly on compiler version, processor
capabilities, etc. Branch prediction, code cache misses, virtual function
inlining and caches are topics better served for a discussion on
optimization not design. The code in [1] produces assembly output by where
we inline the virtual function without optimization( -O0)[2], even on a
portable arch setting in GCC; however, the same code produces a different
assembler output on gcc 4.5.3 with optimizations enabled [3]. If I run the
optimized code, the vtable segment is approximately two nanoseconds faster
than the inlined code because it results in smaller code for better branch
prediction and instruction cache hits. Perf stat shows this across 1M runs,
and in a more illustrative example, we can see how jumps aren't always
slower than inlining. A great example of this exists in last year's CPPCON
from Chandler Carruth. My point is not to argue, but rather to suggest that
cons such as this have such wildly varying characteristics it's akin to
suggesting anyone really knows what the Java Garbage collector is going to
do at any given point [4].

Nonetheless I'm suggesting we only inject and cache a function that's run
to evaluate. This certainly may require an extended FlowConfiguration and
changes to Property, ProcessorNode, and ProcessorContext, but the changes
should hopefully minimize EL specific code on the core so that if/when we
move certain parts to C for 'librarification' we can do so more easily.
Additionally, if there is ever the need to completely disable EL in
production, we can do so by changing the injected classes within the
binaries via C2.

Of course, these are only suggestions. I would eventually like to see
someone who is using libminifi as a library be able to inject this in the
same way, but not be required to include it if they have only the need to
do a very small task. For the same reason I've been slow to respond, I
unfortunately responded to this very quickly so if I didn't make any sense
please let me know [5].

[1]https://gist.github.com/phrocker/100a0806fbc188731358b659c7cf63ab

[2]

leaq -48(%rbp), %rax
addq $8, %rax
movl $.LC0, %esi
movq %rax, %rdi
call std::basic_string::operator=(char const*)
leaq -48(%rbp), %rax
addq $8, %rax
movq %rax, %rdx
leaq -64(%rbp), %rax
movq %rdx, %rsi
movq %rax, %rdi
call std::basic_string::basic_string(std::basic_string const&)
leaq -64(%rbp), %rax
movq %rax, %rdi
call std::basic_string::~basic_string()

[3]

subq $64, %rsp
leaq 16(%rsp), %rbx
movq vtable for Property2+16, 16(%rsp)

[4]
  haha.

[5]
  I usually don't.

On Thu, Nov 9, 2017 at 11:06 AM, Andy Christianson 
wrote:

> MiNiFi - C++ Devs,
>
> I am currently working on MINIFICPP-49, the expression language feature.
> While expression compilation and evaluation is fairly self-contained, at
> the very least the API to access expression evaluation will touch core
> components.
>
> Here is how NiFi is currently exposing expression evaluation to processors:
>
> ...
> try {
> // read the url property from the context
> final String urlstr = trimToEmpty(context.getPropert
>