Re: MiNiFi C++ chashes with Remote Process Group

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Mauro,

In addition to what Marc said, please run minifi under gdb and run "bt" to
provide a backtrace when the process crashes.

Regards,

Andy I.C.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJa6zD8AAoJEG1+mBKNMpIDTdUH+gOkEVQWKlqGVsnJCjXviEmx
j5eL0cdAtq/8NdJ1ZW1c/mfyGDX18UzJSLZhHVknkqbbQhNYQbjcIzNpt9Lqz0k8
97KikyX592SgqdaTN+2CENDc/FNur9TPvZYqs2047RRMXoSUK+fqmZXA3OydL8Nm
Xse/30IoGJIApxYQDvTlG2OXf4Tdj828Tnhc/rtrhTfJY1Dt/Fo/uRWp3XViX4Oi
vzH9OZDM/W4vSSD7xTDEODTQBuhjjOo5NNgzB2J25fojAPoK7zSjb/JQtIk0ZJP6
oo7niUZKVMyrSWINUgo17KXKFZqURQOqji/Onv9S7BvzUSSP0n22DagtmlczdeY=
=J268
-END PGP SIGNATURE-



​Sent from ProtonMail, Swiss-based encrypted email.​

‐‐‐ Original Message ‐‐‐

On May 3, 2018 9:39 AM, mauro  wrote:

> ​​
> 
> Dear community,
> 
> I am testing MiNiFi C++ with an embedded device running the Yocto
> 
> distribution. So far I managed to compile it and run a simple flow file
> 
> "getfile->putfile" on the same device.
> 
> However when I add a Remote Process Group to the flow MiNiFi crashes with
> 
> segmentation fault, see logs below.
> 
> I have increased all the log properties to Debug but still I cannot see
> 
> where the problem is.
> 
> root@sbc-lynx:~/nifi-minifi-cpp-0.4.0# bin/minifi.sh run
> 
> [2018-05-infol 13:10:39.374] [main] [info] Using
> 
> MINIFI_HOME=/home/root/nifi-minifi-cpp-0.4.0 from environment.
> 
> [2018-05-infol 13:10:39.376] [org::apache::nifi::minifi::Properties] [info]
> 
> Using configuration file located at
> 
> /home/root/nifi-minifi-cpp-0.4.0/conf/minifi-log.properties
> 
> setting default dir to /home/root/nifi-minifi-cpp-0.4.0/content_repository
> 
> bin/minifi.sh: line 257: 1489 Segmentation fault ${minifi_executable}
> 
> I have attached the minifi-app.log as well.
> 
> The same flow works if I run it on a Windows machine after amending the
> 
> paths for get and put file.
> 
> Could you please advise what to look for this issue?
> 
> Thanks in advance,
> 
> Regards,
> 
> Mauro
> 
> minifi-app.log
> 
> http://apache-nifi-users-list.2361937.n4.nabble.com/file/t440/minifi-app.log
> 
> --
> 
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

RE: [NIFI-CPP] Issues with TIMER_DRIVEN and CRON_DRIVEN

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Iyán,

The scheduling for GetUSBCamera is a bit different. The docs (PROCESSORS.md)
say it "outputs one frame per flow file" --it should probably say "one flow
file per frame."

The reason this processor is a bit different is that the underlying camera
library (libuvc) registers a callback that we provide, which is called back at
the given FPS. In this callback, we generate and transfer a new flow file.

This is done so that the flow file outputs are synchronized with the frame rate
of the camera. This reduces latency and allows us to avoid needing a temporary
buffer for onTrigger to copy from. The downside is that the configurating is
different (scheduling is affected only by FPS, nothing else).

The onTrigger function of GetUSBCamera does nothing, so the usual scheduling
mechanisms will do nothing.

As for the image dimensions/quality, GetUSBCamera will enumerate all supported
modes for the camera and select the highest dimensions/quality for the given
FPS. The dimensions are dictated by the camera hardware/drivers, and not us. In
order to change the dimensions after collecting frame data, an additional
expensive resize operation would need to be performed, so it may be better to
have that explicitly done in a follow-on processor. This is something we could
look at changing in GetUSBCamera, but we should carefully consider the
efficiency and configuration UX. A possibility might be to let the user specify
a preferred format in the config, and select the closest supported by the
hardware/drivers. We could also look at supporting resize in the processor, but
that has some downsides (mainly efficiency).

If you are using the images as inputs to a TensorFlow graph, a good place to
resize may be at the start of the TensorFlow graph itself. That way the resize
operation can be parallelized and possibly hardware accelerated.

I have added a ticket [1] to clarify this in the documentation.

Regards,

Andy I.C.

[1] https://issues.apache.org/jira/browse/MINIFICPP-464
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJa3fU6AAoJEG1+mBKNMpID16cIAK7Tk3ykSzCE/9TLb2dLvOyf
jWd0Ced/KhjKBBA+eN6iHxe4eYs/gQtP+Rix+kP93gLTPrZ7vF4Cxb3jT3zra6pI
DaCxp6zUzcGtd960XXnQ1wEii1rIyamokuK8A4uIUFKgcZK6WKgRlcGr6lOWQ5c3
J6/VJNb/eL076lDq6Xj4zU0msxsTMUZWBXJjZ5K6JlI+Y983u+bfNKqnUArWgt/z
KIp8DIBXnib7xNQAIrk8k7THZkS+0/SENSt9j8OV/P2rkXUaWmBwFSX1F4s+k9Dq
4iUvga9mSwvuThR7IiJqrcDJjwo09qHWOVAoTPkzcjYlIGGshUOeIoHcmWKJV24=
=3esw
-END PGP SIGNATURE-



​Sent from ProtonMail, Swiss-based encrypted email.​

‐‐‐ Original Message ‐‐‐

On April 23, 2018 10:35 AM, Mendez Veiga, Iyan 
 wrote:

> ​​
> 
> Ok, thanks Marc. I have tried your file (just making it simpler and adding a 
> RPG) and timer does not work for me... I will send the log later when I 
> arrive home.
> 
> So, just a silly question. Imagine I want a picture every 10 seconds. I would 
> just have to change scheduling period to 10 sec or 1 msec, right?. Does 
> this work even if I specify FPS: 24?
> 
> I noticed that the video profile is chosen based on the FPS value. If I set 
> FPS: 1 I get very large images while with FPS: 24 I get smaller 800x600 
> images. I guess this will be configured as a property in the future.
> 
> Best regards,
> 
> Iyán
> 
> -Mensaje original-
> 
> De: Marc P. [mailto:marc.par...@gmail.com]
> 
> Enviado el: lunes, 23 de abril de 2018 13:16
> 
> Para: users@nifi.apache.org
> 
> Asunto: Re: [NIFI-CPP] Issues with TIMER_DRIVEN and CRON_DRIVEN
> 
> This Message originated from a Non-ArcelorMittal source
> 
> Iyán,
> 
> Yes, both strategies seem to work.
> 
> Here is my config file
> 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_phrocker_a209b5163044a5bb3c2e6ac4662ce143=DwIFaQ=y5LGzd1hT50ruE_IlUH7x8VGgWz9W0tFVWT6rSvPUKA=OqJOXvZGIFafgLt7Uocjlzsiwprj2_HJJ0p7HG9IsupasVjguMhbhoUfzdA5V_Qn=CCOIqqrapqve7SfBS7uoDgFCrVh83PSU7mN-ZblHnho=ehLOY9yBJ2LpM0Jnl-xWpiL5r35oVQnvoglF9U_m118=
> 
> Below is the output form my pi. Due to the time it takes to create a PNG and 
> the scheduling strategy it usually ends up being about a second between 
> pictures. When I reduce my configuration to what you have ( for the timing 
> config options ) I maintain the same pattern of taking a picture every 900 
> ms. Logs may help further diagnose the issue.
> 
> [2018-04-23 11:11:17.484]
> 
> [org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
> 
> [2018-04-23 11:11:18.690]
> 
> [org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
> 
> [2018-04-23 11:11:19.850]
> 
> [org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
> 
> [2018-04-23 11:11:20.998]
> 
> [org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
> 
> [2018-04-23 11:11:22.145]
> 
> [org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
> 
> [2018-04-23 11:11:23.284]
> 
> 

Re: [NIFI-CPP] Problem with UpdateAttribute processor

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Iyán,

It looks like you are using the 0.4.0 release tar, which does not include
UpdateAttribute. Please try building with the latest master release.

Regards,

Andy I.C.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJa2OKZAAoJEG1+mBKNMpIDfoUH/1S7OxNz6Y2QIs/rmAAgQbrb
CjBa0FYUWw7UNp6n+Jxze8pnlr7t/WrYDg0nhJAoDQTKN0QtSK4rQ2Uz/NVCdAuA
mgptc0IG0ob6xGk8kF9emAxCBeBIc1o/7bte9ynkzaOcFrz6UGkMIjb9bj+lChlW
1Ls9AJWofwp+gy76qe7K6f7lm+NTTbzTQ/2ELYyCMhGwlrg3ozo6E2PlPLCDgIPg
aJFT+V4QY0ZEk8T44AL+FHMf3hmq5wV5GP/cY8wFvYqtIYnm+1aJmcOwSZf3nhfw
z4QEXjYSCPnxa10hdmf1Xj7xP1DgRl7elwXg86GFdLBaBcrZrLfUHbPFDD3taDo=
=ehbS
-END PGP SIGNATURE-



​Sent from ProtonMail, Swiss-based encrypted email.​

‐‐‐ Original Message ‐‐‐

On April 18, 2018 7:51 PM, Iyán Méndez Veiga  wrote:

> ​​
> 
> I attach here another log file with TRACE level instead of INFO. Hope it
> 
> helps.
> 
> Best regards,
> 
> Iyán
> 
> 
> -
> 
> Iyán Méndez Veiga | Physicist
> 
> GPG: 0x422E3694311E5AC1
> 
> Web: https://iyanmv.com
> 
> ♫♪.ılılıll|̲̅̅●̲̅̅|̲̅̅=̲̅̅|̲̅̅●̲̅̅|llılılı.♫♪

Re: [NIFI-CPP] Problem with UpdateAttribute processor

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Iyán,

UpdateAttribute should be fairly straightforward, as it's a core processor.
This should be a simple issue to figure out.

We're looking into this.

Regards,

Andy I.C.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJa2NETAAoJEG1+mBKNMpIDF2MIAK0EAE+Prjl+h8nfHKNsjjoQ
5K5E0fy8vdopFuQ2eefmsBnttG/5Nl1oBe2fsg2l1k0/HkF5KLIH3wH7LCZjoTcd
cexe2MAwiSliBECT7VtIFeWtNMYHn+eQw0niIF14TfSPcQ/6bGDEuRiVpI4bJAeI
PUhzFneNzHqECaZlTNfzbABoD4bEyosIqpG3l0d85fMVd4UjQK4U8n+5ZJe+ymdY
0HgnvsWTilpaP0aDuHjh8M8HpJX5L84BtemrWQmm5KTsih2qsvcfw2vZXvpX3pZ0
PuJ9bBmrgNkfcwgQVfPeUznyv/NlNkyPLVlMqn5WiiO7WXxrDiLR7i5dfuvyDMw=
=M2U2
-END PGP SIGNATURE-



​Sent from ProtonMail, Swiss-based encrypted email.​

‐‐‐ Original Message ‐‐‐

On April 18, 2018 7:24 PM, Iyán Méndez Veiga  wrote:

> ​​
> 
> Hi,
> 
> After few days trying NiFi and MiNiFi C++ I (finally) have been able to
> 
> compile minifi-cpp in a rasp3 with tensorflow and usb camera support.
> 
> Now I am having a silly issue. I cannot use UpdateAttribute processor
> 
> successfully. This is the error I get:
> 
> [org::apache::nifi::minifi::core::FlowConfiguration] [error] No Processor
> 
> defined for UpdateAttribute
> 
> I attached the relevant files in the email.
> 
> What I am trying to do is replicate what Andy explains in this article:
> 
> https://community.hortonworks.com/articles/174520/minifi-c-iot-cat-sensor.html
> 
> Of course, mini is a silly example. But using this config file:
> 
> https://gist.github.com/achristianson/1dea217e5fcbc88b87e526d919dad2c0#file-config-yml-L93
> 
> I get the same error in the step where tf.type attributes are added to the
> 
> DataFlows.
> 
> Hope you can help me.
> 
> Best regards,
> 
> Iyán
> 
> 
> -
> 
> Iyán Méndez Veiga | Physicist
> 
> GPG: 0x422E3694311E5AC1
> 
> Web: https://iyanmv.com
> 
> ♫♪.ılılıll|̲̅̅●̲̅̅|̲̅̅=̲̅̅|̲̅̅●̲̅̅|llılılı.♫♪

RE: USB Camera support with MiNiFi 0.4.0 in Rasp3?

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Iyán,

Our FindTensorFlow.cmake file is used during the build to locate the TensorFlow
native libraries. It checks in /usr/lib, /usr/local/lib, and
/usr/local/lib/tensorflow_cc. If the libraries are not present at any of those
locations, a hint can be given using the TENSORFLOW_PATH variable.

This will hopefully work for you:

cmake -DTENSORFLOW_PATH=

Also, the tensorflow_cc [1] project (which installs to a location
FindTensorFlow.cmake searches) has worked well for us in the past and may save
compilation time since it only builds the needed libraries.

Regards,

Andy I.C.

[1] https://github.com/FloopCZ/tensorflow_cc
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJa1LoxAAoJEG1+mBKNMpIDOikIAI9IGS96Sj8bxlzuuHl3L30S
0ySNv9zQPKQNwsulpIQZXGiF2zXt/GK8O5GcMFfED7WXEcvKtEZDlF8MUgLxG4pa
31Q3me/qIJZHsUukXOZ8bMCxTrfS3ljiXhyJACpHRbnJGuZX3zAkjRF3NUYQBA/m
Zzvf8bSsAYObwp8txbzUtd+9u3+GWGJicSPRus3T/k82Zw8sCh0l433ExEWjdAxk
VaXGzjKwmCkMgZnF+rtyiTgEq7v31h+Dbdc1PI3zuPBqJz3NOK9JN3rz44XnuMKc
lYM7XUfhqiFAriU7BzsRfVRbfA0lVxsqPuh8QaL9ntQ0FGVN/c87L3OvYlXeneU=
=0Ims
-END PGP SIGNATURE-


​Sent from ProtonMail, Swiss-based encrypted email.​

‐‐‐ Original Message ‐‐‐

On April 13, 2018 2:47 AM, Mendez Veiga, Iyan 
<iyan.mendezve...@arcelormittal.com> wrote:

> ​​
> 
> Hi Mark, Andy,
> 
> I was able to compile TensorFlow in the rasp3 using gcc 4.8.5, bazel 0.10.1 
> and TensorFlow 1.6.0 (other combinations failed at some point). However, when 
> I try to build MiNiFi C++ 0.4.0 with -DENABLE_TENSORFLOW=TRUE, it fails to 
> find it. Should I define some env variables or do something else?
> 
> Thanks,
> 
> Iyán
> 
> -Mensaje original-
> 
> De: Marc P. [mailto:marc.par...@gmail.com]
> 
> Enviado el: lunes, 09 de abril de 2018 17:23
> 
> Para: users@nifi.apache.org; Andy Christianson aichr...@protonmail.com
> 
> Asunto: Re: USB Camera support with MiNiFi 0.4.0 in Rasp3?
> 
> This Message originated from a Non-ArcelorMittal source
> 
> You may need to install libusb-1.0-0-dev. This is part of the reason
> 
> why it was disabled for aptitude distros in 0.4.0; however, with this
> 
> added to my bootstrap branch I've successfully installed and run
> 
> GetUSBCamera on the RPi and other Ubuntu distros ( 32 and 64 bit too )
> 
> On Mon, Apr 9, 2018 at 11:17 AM, Andy Christianson
> 
> aichr...@protonmail.com wrote:
> 
> > -BEGIN PGP SIGNED MESSAGE-
> > 
> > Hash: SHA1
> > 
> > Iyán,
> > 
> > In addition to what Marc said, IIRC, libuvc and libusb are the main
> > 
> > dependencies that need to be available. The libuvc library is included in
> > 
> > thirdparty/ and can be build along with minifi. The libusb-dev package is
> > 
> > needed for the build.
> > 
> > As long as those two libraries are built/available, things should build. You
> > 
> > will need to get identifiers for the USB device (using, e.g. lsusb). 
> > Additional
> > 
> > info about this should be in PROCESSORS.md.
> > 
> > Regards,
> > 
> > Andy I.C.
> > 
> > -BEGIN PGP SIGNATURE-
> > 
> > Version: GnuPG v2.0.22 (GNU/Linux)
> > 
> > iQEcBAEBAgAGBQJay4P3AAoJEG1+mBKNMpIDNHMH/2dUgrbXKME7b9qVWTwjcRvQ
> > 
> > FbsOf9mgHTAhQHtAGDWDSIfnYNhkoLzuuJeQREitBkInA9Obpw84IIXfkWlIDM3Z
> > 
> > 6enByZ4l9+tHBARu7I5/iGmoWk/suC9ohTweOfSGvHeZL0r9+P7llWHF6pIk4h97
> > 
> > 8uhVERv5Qw84g31bGIM+h/1JpTFMQqFNjnVrtKYY+uCSOicXKCY3uwY1I88c5iXZ
> > 
> > M3kkyYhBAiXt4oFkdIJp0xWCyJvFmkC9cthyy8f7BCMgV6Z7EgffScybYUgUo81Y
> > 
> > L6yYTXJetqKHNDf4u7wDQBfl8iX3FfPqAwn1r5JhJ0IRj3tBkmyTufziTS8Twss=
> > 
> > =dQ7j
> > 
> > -END PGP SIGNATURE-
> > 
> > Sent from ProtonMail, Swiss-based encrypted email.
> > 
> > ‐‐‐ Original Message ‐‐‐
> > 
> > On April 9, 2018 8:35 AM, Marc P. marc.par...@gmail.com wrote:
> > 
> > > Iyan,
> > > 
> > > That one along with a few others required a bit more manual effort
> > > 
> > > at the time in bootstrap and thus need to be enabled manually. Enable
> > > 
> > > that one manually with cmake -DENABLE_USB_CAMERA=1 ..
> > > 
> > > In the next version it'll be added to the bootstrap with 0.5.0. I
> > > 
> > > was running GetUSBCamera on a Pi last night but I don't specifically
> > > 
> > > recall if there was much more to take into account. I have a branch
> > > 
> > > where USB Camera can be enabled on PI. I'll test that on a Pi and
> > > 
> > > submit a PR if it w

Re: USB Camera support with MiNiFi 0.4.0 in Rasp3?

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Iyán,

In addition to what Marc said, IIRC, libuvc and libusb are the main
dependencies that need to be available. The libuvc library is included in
thirdparty/ and can be build along with minifi. The libusb-dev package is
needed for the build.

As long as those two libraries are built/available, things should build. You
will need to get identifiers for the USB device (using, e.g. lsusb). Additional
info about this should be in PROCESSORS.md.

Regards,

Andy I.C.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJay4P3AAoJEG1+mBKNMpIDNHMH/2dUgrbXKME7b9qVWTwjcRvQ
FbsOf9mgHTAhQHtAGDWDSIfnYNhkoLzuuJeQREitBkInA9Obpw84IIXfkWlIDM3Z
6enByZ4l9+tHBARu7I5/iGmoWk/suC9ohTweOfSGvHeZL0r9+P7llWHF6pIk4h97
8uhVERv5Qw84g31bGIM+h/1JpTFMQqFNjnVrtKYY+uCSOicXKCY3uwY1I88c5iXZ
M3kkyYhBAiXt4oFkdIJp0xWCyJvFmkC9cthyy8f7BCMgV6Z7EgffScybYUgUo81Y
L6yYTXJetqKHNDf4u7wDQBfl8iX3FfPqAwn1r5JhJ0IRj3tBkmyTufziTS8Twss=
=dQ7j
-END PGP SIGNATURE-


​Sent from ProtonMail, Swiss-based encrypted email.​

‐‐‐ Original Message ‐‐‐

On April 9, 2018 8:35 AM, Marc P.  wrote:

> ​​
> 
> Iyan,
> 
> That one along with a few others required a bit more manual effort
> 
> at the time in bootstrap and thus need to be enabled manually. Enable
> 
> that one manually with cmake -DENABLE_USB_CAMERA=1 ..
> 
> In the next version it'll be added to the bootstrap with 0.5.0. I
> 
> was running GetUSBCamera on a Pi last night but I don't specifically
> 
> recall if there was much more to take into account. I have a branch
> 
> where USB Camera can be enabled on PI. I'll test that on a Pi and
> 
> submit a PR if it works.
> 
> Thanks,
> 
> Marc
> 
> On Mon, Apr 9, 2018 at 3:46 AM, Mendez Veiga, Iyan
> 
> iyan.mendezve...@arcelormittal.com wrote:
> 
> > Hi,
> > 
> > I am trying to compile MiNiFi C++ 0.4.0 in a raspberry pi 3 running 
> > Raspbian.
> > 
> > I have installed all packages in the requirements (and some optional ones, 
> > too):
> > 
> > https://github.com/apache/nifi-minifi-cpp#system-requirements
> > 
> > However, when I execute the bootstrap script I cannot enable the USB Camera 
> > support (since I want to use GetUSBCamera processor) and the only 
> > information I get is that "Extension cannot be installed due to version of 
> > cmake or other software".
> > 
> > I couldn't find any information on the Internet. Any one is facing a 
> > similar issue?
> > 
> > Regards,
> > 
> > Iyán

Re: MiNiFi C++ and Tensorflow future plans?

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Iyán,

> I was wondering what is the roadmap and future plans of MiNiFi C++ agent
> regarding Tensorflow processors. With the three that are mentioned in the
> article it is possible to classify images on edge but I would like to know if
> other processors from TF will be included so we can train a neural network on
> edge.
> 
> Also, if you share with me a list of features that will be implemented in the
> near futures, it would be very helpful.

Glad to hear you are interested in the project. The current plan is to cover
the common edge inference use-cases. This includes general ML inference tasks
as well as computer vision tasks including object classification, object
detection (multiple output FlowFiles with bounding boxes + class),
classification/anomaly detection in log files, time series anomaly detection
(think temperature sensors).

The benefit of using MiNiFi - C++ rather than just a pure TensorFlow model is
that all of the usual NiFi techniques like routing on attribute (i.e. object
class or other ML-inferred metadata), sending to cloud endpoints (S2S to a NiFi
instance, or ingest into a Kafka or MQTT queue, etc.), and arbitrary scriptable
actions (ExecuteScript processor) are all fairly simple to do.

MiNiFi - C++ is a community-driven Apache project, so it ultimately will
include any FlowFile -> tensor or tensor -> FlowFile processor that is
developed to satisfy community requirements.

What use cases do you have in mind? MiNiFi - C++ is nearing its 1.0 release and
as such most common use cases, including deep neural networks or even training
on the edge (given sufficient resources, i.e. GPU) should be possible without
too much custom programming effort. We'd like to get your input/feedback as a
potential community participant so that MiNiFi will become more useful to
everyone over time.

Regards,

Andy I.C.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJaxjuRAAoJEG1+mBKNMpID8AIH/3qpZXhOPZOWoXwxn+RUJeBz
eoFFFw9QjCUhhgUeyOUAe1YH4EaMBjE0qewrBhZrybBxjYgvaOs7GGjqrTpbZmf6
1vhRUTqzNeI68ke2+tjk0xCtzTQd3Ag92kpBLQn4y6CshUS9YSvXzbPjXatvx2gw
w6uKR+Y9uTplf9UIxWXAUdgL1vPLPaXy5OgiUwIflfd4Eb5OvS3Dh8yS6MhfzjcF
Zfwn84Op+gtJQr2YJTOH/dgOiQC1SPpR8RYsme5QcQIZ6qfcR2SIfMD70nixQi4W
RXlj9Ot3crV0a0hEnFBWfhO/UWTagwM/isMD543RBdmbGcYUoTWnVH/Wau7GMJ0=
=VJFY
-END PGP SIGNATURE-


​Sent from ProtonMail, Swiss-based encrypted email.​

Re: minifi on android

[Andy Christianson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Mike,

I have successfully compiled a static MiNiFi - C++ binary/library natively for
Android. The build requires some changes (mainly external builds for some
dependencies we don't already bundle and some cmake adjustments).

The only requirements on your host, other than the standard Android SDK/NDK
tooling, is flex & bison (required for expression language --you can disable
expression language to remove this dependency, or use a compatible lexer/parser
generator).

The patches required to make this work need to be cleaned up and turned into
proper pull requests. This is something that I'm planning to do in the near
future. If you're interested, I can send you the full patch if you want to
contribute the PRs, or just apply it for your own (unofficial/unsupported) use.

Regards,

Andy I.C.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJaxOwoAAoJEG1+mBKNMpIDd08H/RZfjwwcliSoab/DmN90d6M6
/KX0UkJVQJdhGnxlz3CCntoG/ADc/iG2w49F7upUCFJFUNHkwyrve2Zz48pAlbut
ystA5y+mmaZBLabffPh7Gq8bpHlAUZwYOt+jCVJchZXNg6bccWktf+0y2UkIBXok
/VlRGh9rFtZsLo3wb5nb52Jbj3i4+eWjHrVRw7SVa4xkkSgBgooNA5VqkgopDuM8
1yeTuKg+XYjM3Iuu6aton1Pikh/KxY7/bTcZ7KyfDe2x2NrH9XsMXrq7A6EAId7E
dLum7fev5ropNH0cdQBm7zrw8xaPbklg7I7gm2OBKmWUXkaDjAEnfTdIXuuAX9c=
=FYcV
-END PGP SIGNATURE-

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.

‐‐‐ Original Message ‐‐‐
On April 4, 2018 10:51 AM, Michael Riddle  wrote:

> Hi Folks,
>
> Based on this page: 
> https://community.hortonworks.com/questions/34320/has-anyone-gotten-apache-nifi-running-on-an-androi.html.
>  It should be possible to cross-compile minifi using the android NDK.
>
> A couple of questions here:
> Has anyone successfully done this that you are aware of? If so, are the 
> additional necessary steps detailed somewhere?
>
> On my CentOS 7 box, I've gotten as far as getting the bootstrap running with 
> an android-27 standalone-toolchain using the default gnustl, but running make 
> immediately fails complaining that it can't find FlexLexer.h. When I edit the 
> following lines in CMakeCache.txt to look in /usr/include for header files:
> CMAKE_CXX_FLAGS:STRING=-isystem /usr/include
> CMAKE_C_FLAGS:STRING=-isystem /usr/include
>
> It blows up with:
> /opt/my-android-toolchain/include/c++/4.9.x/aarch64-linux-android/bits/ctype_base.h:36:2:
>  error: #error Bionic header ctype.h does not define either _U nor _CTYPE_U
>  #error Bionic header ctype.h does not define either _U nor _CTYPE_U
>
> Will I need to recompile flex, bison, etc. all as android to be able to use 
> them, or should they "just work"?
>
> Thanks,
> -Mike Riddle

Re: updateAttribute fails with hostname(true) in minifi-cpp

[Andy Christianson]

Tom,

The current hostname() implementation supports no arguments. Support for the 
FQDN argument will come with 
https://issues.apache.org/jira/browse/MINIFICPP-368.

The error message you're getting could certainly be improved. It could be that 
we have a bug with regard to subjectless functions with arguments. If that's 
the case, it will come to light when MINIFICPP-368 is completed (this ticket is 
on the near-term plan).

Regards,

Andy I.C.

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.

‐‐‐ Original Message ‐‐‐
On March 29, 2018 3:37 AM, Tom Burton  wrote:

> I have had minifi-cpp failing to go beyond an updateAttribute processor if I 
> have a dynamic property being polulated with the FQDN using 
> ${hostname(true)}.  It throws an error saying it is expecting a “)” or a “,”. 
>  It works fine when just using ${hostname()}.  This is running on an 
> environment without a domain controller and the endpoint does not have an 
> entry in the dns, so I would expect fqdn to provide the same result as a 
> simple hostname.  When interrogating
>
> # hostname –fqdn
>
> it provides the same response as
>
> # hostname
>
> I’m running it on Centos7
>
> Yours, Tom

Re: minifi-cpp crashes ssh

[Andy Christianson]

Tom,

It sounds like the stop script might be picking up the wrong PID. To verify 
this, would you mind sending the exact command you're running, plus full log 
output (console, plus all MiNiFi logs)?

Regards,

Andy I.C.

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.

‐‐‐ Original Message ‐‐‐
On March 29, 2018 2:54 AM, Tom Burton  wrote:

> It may be stupid operator error but I have noticed that if you try to stop 
> minifi-cpp when it isn’t running it crashes the ssh daemon.  I don’t know 
> what else it impacts but it would be useful to have some exception handling 
> so it checks whether an instance it running before taking any action.  I’ve 
> observed this when running on Centos7 built from the git master a couple of 
> days ago.
>
> Tom

RE: [EXT] Re: Polling Processors impact on Latency

[Andy Christianson]

Chirag,

Peter's note about bored yield duration is right on. Some additional things I'd 
like to point out are:

1) You might get the lowest latency in a configuration where the processor runs 
continuously (bored yield duration 0). This is because with the code executing 
continuously, CPU caches should stay hot. The trade-off is wasted CPU cycles 
when the consumer is waiting for input more often than processing it.

2) We do have an event driven scheduling mode. For NiFi this still appears to 
be experimental:

"Event driven: When this mode is selected, the Processor will be triggered to 
run by an event, and that event occurs when FlowFiles enter Connections feeding 
this Processor. This mode is currently considered experimental and is not 
supported by all Processors. When this mode is selected, the ‘Run schedule’ 
option is not configurable, as the Processor is not triggered to run 
periodically but as the result of an event. Additionally, this is the only mode 
for which the ‘Concurrent tasks’ option can be set to 0. In this case, the 
number of threads is limited only by the size of the Event-Driven Thread Pool 
that the administrator has configured."

https://nifi.apache.org/docs/nifi-docs/html/user-guide.html

This mode is also implemented in MiNiFi - C++ and is done using condition 
variables. This type of event-driven scheduling should put you near the lower 
limit of latency without sacrificing much CPU, assuming a workload where the 
consumer is waiting for input more often than processing it.

I would suggest trying out different configurations and taking measurements, as 
the ideal config will depend a lot on your workload.

Regards,

Andy I.C.

Sent from ProtonMail, Swiss-based encrypted email.


> Original Message 
>Subject: RE: [EXT] Re: Polling Processors impact on Latency
>Local Time: November 7, 2017 7:29 AM
>UTC Time: November 7, 2017 12:29 PM
>From: pwi...@micron.com
>To: users@nifi.apache.org 
>
>
>If you schedule the processor to run every 0 sec (the default) then in my 
>experience you won’t notice latency from polling at all. But I guess this 
>depends on your expectations,
> volume, and over all Flow processing time.
>
>
>
>Yes, event driven may help, but from what I’ve read it’s more about reducing 
>server resource consumption than latency (could be wrong).
>
>
>
>As for a hard set limit, there is a configuration entry in nifi.properties 
>that seems relevant:
>
>
>
># If a component has no work to do (is "bored"), how long should we wait 
>before checking again for work?
>
>nifi.bored.yield.duration=10 millis
>
>
>
>Thanks,
>
>  Peter
>
>
>
>From: Chirag Dewan [mailto:chirag.dewa...@yahoo.in] 
>Sent: Tuesday, November 07, 2017 8:02 PM
>To: apere...@gmail.com; users@nifi.apache.org
>Subject: [EXT] Re: Polling Processors impact on Latency
>
>
>Thanks Andrew for the quick response.
>
>
>I am more concerned about the processors polling for flow files on the 
>connection between the processors?
>
>Thanks,
>
>Chirag
>
>Sent from Yahoo Mail on Android
>
>
>>On Tue, 7 Nov 2017 at 5:24 PM, Andrew Grande
>> wrote:
>>Yes, polling increases latency in some cases. But no, NiFi is not just 
>>polling. It has all kinds of sources, and listening vs polling vs subscribing 
>>purely depends on the protocol of that given processor.
>>
>>Hope this helps,
>> Andrew
>>
>>
>>
>>On Tue, Nov 7, 2017, 1:39 AM Chirag Dewan  wrote:
>>>Hi All,
>>>
>>>I am a layman to NiFi. I am exploring NiFi as a data flow engine to be 
>>>integrated with my Flink processing engine. A brief history of our approach 
>>>: 
>>>
>>>We are trying to build a Streaming Data processing engine. We started off 
>>>with Flink as the sole core engine, which is responsible for 
>>>collection(through Flink Sources) as well as processing
>>> the data. 
>>>
>>>Soon we fumbled onto NiFi and the data flow world. 
>>>
>>>So far, my understanding is that the NiFi processors are poling processors 
>>>and not Pub-Sub processors. That makes me wonder, whats the impact of 
>>>polling on latency? I know I can configure
>>> my processors to tradeoff latency with throughput, but is there a hard set 
>>> limit on the latency I can achieve using NiFi? 
>>>
>>>As I said, I am layman as yet. Perhaps my understanding is short here. Any 
>>>leads would be much appreciated. 
>>>
>>>P.S - Not diving much into Event Driven Processors. They look like something 
>>>which might clear my thoughts. But since they are marked experimental, would 
>>>be more interested in understanding
>>> the timer driven processors.
>>>
>>>Thanks,
>>>
>>>Chirag
>>>
>>>

Re: Flow File as Input to ExecuteScript Processor

[Andy Christianson]

Glad to hear you got it working, and documented the working script for future 
reference.

Regards,

Andy I.C.


From: rakesh 
Sent: Wednesday, October 25, 2017 1:37 AM
To: users@nifi.apache.org
Subject: Re: Flow File as Input to ExecuteScript Processor

Hi Andy,


Thanks for your inputs

Finally my problem solved, below is my script.

WORKINGstringrevExample.txt



Thanks
Rakesh.



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Flow File as Input to ExecuteScript Processor

[Andy Christianson]

Rakesh,

The ExecuteScript Cookbook series [1] by Matt Burgess is a great introduction 
to scripting in nifi.

The essence of it is that you need to use the session object (which is provided 
for you automatically) to get a flow file:

flowFile = session.get();

Assuming the flowFile is not null/nil/None, you would then use session.write, 
providing your own subclass of StreamCallback, to rewrite the contents 
according to your specifications.

Please note that you do not necessarily need to use groovy. You could use 
python, lua, javascript, or others if you are more comfortable with any of 
those.

Regards,

Andy

[1] 
https://community.hortonworks.com/articles/75032/executescript-cookbook-part-1.html

On 10/18/17, 5:22 AM, "rakesh"  wrote:

Hi All,


I have a JOSN Flow file 


 
 From this  I want to make it like


 

To accomplish this task I selected ExecuteScript Processor using Groovy
script.
I have a regex which matches *Mon Oct 09 23:38:55 IST 2017* and I want to
read the json data and check the regex condition to format is like
*toDate("EEE MMM dd HH:mm:ss z "):toNumber():format("-MM-dd
HH:mm:ss.S")*  and overwrite it to same flow file.

I'm new to Groovy scripting.

Any Suggestions any suggestions would be appreciated.



Thanks
Rakesh.



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Simple check to see whether a file is compressed or not?

[Andy Christianson]

On Wed, 2017-09-20 at 11:32 -0700, Laurens Vets wrote:
> Is there a simple way to see whether a file is compressed or not?
> 
> In my current flow, if an object is not compressed it should bypass the 
> CompressContent processor, otherwise it needs to be decompressed.
> 

Not sure if there is a better way, but one thing you could do is try to
decompress it, and have both the failure and success relationships of
that go to the next processor in your flow.

-Andy I.C.

Re: minifi-cpp s2s no protocol

[Andy Christianson]

> I tested a similar config on the nifi side and no changes were necessary
> to nifi.properties in order to use the HTTP s2s protocol.
> 

This is explained (thanks Aldrin) by
https://issues.apache.org/jira/browse/MINIFICPP-60.

We simply don't support it yet. My test will just cover the socket
protocol for now.

-Andy I.C.

Re: minifi-cpp s2s no protocol

[Andy Christianson]

On Wed, 2017-09-20 at 11:37 -0400, Marc wrote:
> Andy,
>I think you're looking for the Site to Site properties in your
> nifi.properties. You likely have an option named
> "nifi.remote.input.socket.port" as I do, below. Use the input host
> ( probably nifi ) and port you have defined. 
> 
> 
> # Site to Site properties
> 
> nifi.remote.input.host=localhost
> 
> nifi.remote.input.secure=false
> 
> nifi.remote.input.socket.port=10443
> 

That worked. The config had to be set on the nifi side. Doesn't this
meant that it is using/has to use the socket protocol rather than HTTP?
I tested a similar config on the nifi side and no changes were necessary
to nifi.properties in order to use the HTTP s2s protocol.

-Andy I.C.

> 
> On Wed, Sep 20, 2017 at 11:30 AM, Andy Christianson
> <achristian...@hortonworks.com> wrote:
> On Wed, 2017-09-20 at 11:26 -0400, Marc P. wrote:
> > Andy,
> >Can you try adding the RPG host and port in the
> properties for your
> > ports?
> 
> Sure, I'll give that a shot. Since I'm using HTTP protocol on
> port 80,
> Port would be 8080 and Host Name would be 'nifi,' correct?
> 
> -Andy I.C.
> 
> 
> 

Re: minifi-cpp s2s no protocol

[Andy Christianson]

?This is on the phrocker/MINIFI-339 branch with some additional testing code 
I'm working on.


If it helps, in the logs, the s2s connection fails several times as my testing 
nifi instance boots, then once nifi is ready, it gets this config JSON back & 
spits out the no protocol error.


I'm running nifi master branch.


-Andy I.C.


From: Marc <phroc...@apache.org>
Sent: Wednesday, September 20, 2017 11:06 AM
To: users@nifi.apache.org
Subject: Re: minifi-cpp s2s no protocol

Andy,
  What branch are you using?

  I believe this is was one issue resolved under the guise of 
https://issues.apache.org/jira/browse/MINIFICPP-4 ?

On Wed, Sep 20, 2017 at 11:03 AM, Andy Christianson 
<achristian...@hortonworks.com<mailto:achristian...@hortonworks.com>> wrote:

Hi All,


I have a minifi-cpp flow using HTTP s2s to send data to a nifi instance. The 
minifi instance seems to be making an initial connection to nifi, but it fails 
to send data, and I see a "no protocol" in the logs. Here's the log output & 
flow yml:


[2017-09-20 14:37:35.634] [org::apache::nifi::minifi::utils::HTTPClient] [info] 
Submitting to http://nifi:8080/nifi-api/controller
[2017-09-20 14:37:35.648] [org::apache::nifi::minifi::RemoteProcessorGroupPort] 
[debug] controller config 
{"controller":{"id":"96120c9a-b94b-434e-a439-42a4927b5d5e","runningCount":3,"stoppedCount":0,"invalidCount":0,"disabledCount":0,"inputPortCount":1,"outputPortCount":0,"siteToSiteSecure":false,"instanceId":"9fb88d05-015e-1000-f6e9-688ce4c58047","inputPorts":[{"id":"df466fb6-d4f4-4fb1-ac59-5ceb90fcf527","name":"from-minifi","state":"RUNNING"}],"outputPorts":[]}}
[2017-09-20 14:37:35.648] [org::apache::nifi::minifi::RemoteProcessorGroupPort] 
[info] process group remote site2site port -1, is secure 0
[2017-09-20 14:37:35.648] [org::apache::nifi::minifi::RemoteProcessorGroupPort] 
[info] no protocol
?
Connections:
- destination id: dcc4ebf7-e4ae-4e9e-bac5-e0fd11e08033
  name: 86f953f0-6200-448c-b078-3a7c9d082d84
  source id: ea5d3fff-7bc8-4fc7-a01a-7d10013fa883
  source relationship name: success
- destination id: df466fb6-d4f4-4fb1-ac59-5ceb90fcf527
  name: 18a94963-bd92-4043-82c9-1e9e49f2448e
  source id: dcc4ebf7-e4ae-4e9e-bac5-e0fd11e08033
  source relationship name: success
Controller Services: []
Flow Controller:
  name: MiNiFi Flow
Processors:
- Properties:
Input Directory: /tmp/input
  auto-terminated relationships list: []
  class: org.apache.nifi.processors.standard.GetFile
  id: ea5d3fff-7bc8-4fc7-a01a-7d10013fa883
  name: ea5d3fff-7bc8-4fc7-a01a-7d10013fa883
  penalization period: 30 sec
  run duration nanos: 0
  scheduling period: 0 sec
  scheduling strategy: EVENT_DRIVEN
  yield period: 1 sec
- Properties: {}
  auto-terminated relationships list: []
  class: org.apache.nifi.processors.standard.LogAttribute
  id: dcc4ebf7-e4ae-4e9e-bac5-e0fd11e08033
  name: dcc4ebf7-e4ae-4e9e-bac5-e0fd11e08033
  penalization period: 30 sec
  run duration nanos: 0
  scheduling period: 1 sec
  scheduling strategy: EVENT_DRIVEN
  yield period: 1 sec
Remote Processing Groups:
- Input Ports:
  - Properties: {}
id: df466fb6-d4f4-4fb1-ac59-5ceb90fcf527
max concurrent tasks: 1
name: from-minifi
  id: 3a94707d-7717-46fa-b641-57d1104bd927
  name: 3a94707d-7717-46fa-b641-57d1104bd927
  timeout: 30 sec
  url: http://nifi:8080/nifi
  yield period: 10 sec

Ideas?


-Andy I.C.

minifi-cpp s2s no protocol

[Andy Christianson]

Hi All,


I have a minifi-cpp flow using HTTP s2s to send data to a nifi instance. The 
minifi instance seems to be making an initial connection to nifi, but it fails 
to send data, and I see a "no protocol" in the logs. Here's the log output & 
flow yml:


[2017-09-20 14:37:35.634] [org::apache::nifi::minifi::utils::HTTPClient] [info] 
Submitting to http://nifi:8080/nifi-api/controller
[2017-09-20 14:37:35.648] [org::apache::nifi::minifi::RemoteProcessorGroupPort] 
[debug] controller config 
{"controller":{"id":"96120c9a-b94b-434e-a439-42a4927b5d5e","runningCount":3,"stoppedCount":0,"invalidCount":0,"disabledCount":0,"inputPortCount":1,"outputPortCount":0,"siteToSiteSecure":false,"instanceId":"9fb88d05-015e-1000-f6e9-688ce4c58047","inputPorts":[{"id":"df466fb6-d4f4-4fb1-ac59-5ceb90fcf527","name":"from-minifi","state":"RUNNING"}],"outputPorts":[]}}
[2017-09-20 14:37:35.648] [org::apache::nifi::minifi::RemoteProcessorGroupPort] 
[info] process group remote site2site port -1, is secure 0
[2017-09-20 14:37:35.648] [org::apache::nifi::minifi::RemoteProcessorGroupPort] 
[info] no protocol
?
Connections:
- destination id: dcc4ebf7-e4ae-4e9e-bac5-e0fd11e08033
  name: 86f953f0-6200-448c-b078-3a7c9d082d84
  source id: ea5d3fff-7bc8-4fc7-a01a-7d10013fa883
  source relationship name: success
- destination id: df466fb6-d4f4-4fb1-ac59-5ceb90fcf527
  name: 18a94963-bd92-4043-82c9-1e9e49f2448e
  source id: dcc4ebf7-e4ae-4e9e-bac5-e0fd11e08033
  source relationship name: success
Controller Services: []
Flow Controller:
  name: MiNiFi Flow
Processors:
- Properties:
Input Directory: /tmp/input
  auto-terminated relationships list: []
  class: org.apache.nifi.processors.standard.GetFile
  id: ea5d3fff-7bc8-4fc7-a01a-7d10013fa883
  name: ea5d3fff-7bc8-4fc7-a01a-7d10013fa883
  penalization period: 30 sec
  run duration nanos: 0
  scheduling period: 0 sec
  scheduling strategy: EVENT_DRIVEN
  yield period: 1 sec
- Properties: {}
  auto-terminated relationships list: []
  class: org.apache.nifi.processors.standard.LogAttribute
  id: dcc4ebf7-e4ae-4e9e-bac5-e0fd11e08033
  name: dcc4ebf7-e4ae-4e9e-bac5-e0fd11e08033
  penalization period: 30 sec
  run duration nanos: 0
  scheduling period: 1 sec
  scheduling strategy: EVENT_DRIVEN
  yield period: 1 sec
Remote Processing Groups:
- Input Ports:
  - Properties: {}
id: df466fb6-d4f4-4fb1-ac59-5ceb90fcf527
max concurrent tasks: 1
name: from-minifi
  id: 3a94707d-7717-46fa-b641-57d1104bd927
  name: 3a94707d-7717-46fa-b641-57d1104bd927
  timeout: 30 sec
  url: http://nifi:8080/nifi
  yield period: 10 sec

Ideas?


-Andy I.C.