AW: Execute multiple HQL statements in PutHiveQL or SelectHiveQL

2018-04-23 Thread Krstic, Dejan 
Hello Koji,

Yes, I also got the same result in the meantime – records were inserted.

And I completely agree with you. If parser says it “failed to parse” it sounds 
like one big “full-stop” so during my testing I did not even try to check if 
data is actually inserted because message sounds like nothing has been done.

Anyway, thank you for your information and have a nice day! ☺

Br
Dejan

Von: Koji Kawamura [mailto:ijokaruma...@gmail.com]
Gesendet: Donnerstag, 19. April 2018 08:55
An: users@nifi.apache.org
Betreff: Re: Execute multiple HQL statements in PutHiveQL or SelectHiveQL

Hello Dejan,

I tested SET property statements bundled with INSERT statement in a single 
FlowFile passed to PutHiveQL.
The same warning message is logged as you reported. However, actual INSERT was 
successful, I confirmed new rows were inserted.
Please let us know if not the case.

Although the warning is harmless, I agree it can be mis-leading.
I submitted a JIRA and a PR to suppress the warning logs for such SET commands.
https://issues.apache.org/jira/browse/NIFI-5095

Thanks,
Koji


On Tue, Apr 17, 2018 at 5:03 PM, Krstic, Dejan 
> wrote:
Matt,

No, there are no comments at all in this script.

Br
Dejan

Von: Matt Burgess [mailto:mattyb...@apache.org]
Gesendet: Montag, 16. April 2018 18:38
An: users@nifi.apache.org
Betreff: Re: Execute multiple HQL statements in PutHiveQL or SelectHiveQL

Dejan,

Are there any comments in your Hive "script"? If so I believe you need to 
remove them [1].  When I've seen that issue, it's always been because of 
comments before the SET line.

Regards,
Matt

[1] https://issues.apache.org/jira/browse/HIVE-1166


On Mon, Apr 16, 2018 at 12:29 PM, Krstic, Dejan 
> wrote:
Hello,

I have Hive table from which I am trying to load data into other Hive table 
(same server) and to do some transformations in between. The problem is, I need 
to have two SET statements before I actually do INSERT statement and if I try 
to separate everything with semicolons (as configured in processor) I am 
getting following error message:

2018-04-16 16:11:59,368 WARN [Timer-Driven Process Thread-1] 
o.apache.nifi.processors.hive.PutHiveQL 
PutHiveQL[id=9d9290d2-2196-3e7b-a522-ab0e2a46db11] Failed to parse hiveQL: SET 
hive.exec.dynamic.partition = true due to 
org.apache.hadoop.hive.ql.parse.ParseException: line 1:4 missing KW_ROLE at 
'hive' near 'hive'
line 1:8 missing EOF at '.' near 'hive':

The two SET statements at the beginning:

Set hive.exec.dynamic.partition = true;
set hive.exec.dynamic.partition.mode=nonstrict;

I also tried the same thing with SelectHiveQL but it seems not to accept 
multiple queries. Which way would be the best to fix this problem and what 
processor is the best to use in this situation?

Thank you.

Best regards
Dejan Krstic


T-Mobile Austria GmbH
Dejan Krstic
Data Manager

Rennweg 
97-99
A-1030 Wien
E-Mail: mailto:dejan.krs...@t-mobile.at
www.t-mobile.at

DAS VERBINDET UNS.


___

Notice: This e-mail and any attachments are confidential and may be privileged.

If you are not the intended recipient, notify the sender immediately, destroy 
all

copies from your system and do not disclose or use the information for any 
purpose.

Diese E-Mail inklusive aller Anhaenge ist vertraulich und koennte 
bevorrechtigtem

Schutz unterliegen. Wenn Sie nicht der beabsichtigte Adressat sind, informieren 
Sie

bitte den Absender unverzueglich, loeschen Sie alle Kopien von Ihrem System und

veroeffentlichen Sie oder nutzen Sie die Information keinesfalls, gleich zu 
welchem Zweck.



Think before you print!



T-Mobile Austria GmbH

Geschaeftsfuehrung: Dr. Andreas Bierwirth (Vorsitzender), Aufsichtsrat: Dr. 
Rolf Nafziger (Vorsitzender)

Firmenbuch: Handelsgericht Wien, Sitz Wien, FN 171112k, UID ATU 45011703, DVR 
0898295

Konto: UniCredit Bank Austria AG IBAN: AT93 1200 0528 4407 2301, BIC: BKAUATWW



T-Mobile - Das verbindet uns.

___



___

Notice: This e-mail and any attachments are confidential and may be privileged.

If you are not the intended recipient, notify the sender immediately, destroy 
all

copies from your system and do not disclose or use the information for any 
purpose.

Diese E-Mail inklusive aller Anhaenge ist vertraulich und koennte 
bevorrechtigtem

Schutz unterliegen. Wenn Sie nicht der beabsichtigte Adressat sind, informieren 
Sie

bitte den Absender unverzueglich, loeschen Sie alle Kopien von Ihrem System und

veroeffentlichen Sie oder nutzen Sie die Information

OpenID Connect and TLS termination proxy

2018-04-23 Thread Damian Czaja 

Hello,

I'm trying to enable authentication in NiFi using OpenID Connect (with 
Keycloak). I have Keycloak and NiFi behind a TLS termination proxy and a 
reverse proxy, which dispatches the requests to Keycloak and NiFi.


AFAIK, authentication is only possible, if using HTTPS directly on NiFi. 
My problem is that I have a TLS termination proxy, which terminates TLS 
and proxies pure HTTP requests to NiFi. Although I see in NiFi the Login 
button (most probably because the NiFi UI recognize https protocol in 
the browsers URL), but when I click on it, I get a popup with the 
information, that authentication works only in HTTPS.


I tried to set the X-ProxyScheme header to https, but it's still the 
same. Is there any option to have authentication with OIDC on NiFi, when 
NiFi sits behind a TLS termination proxy and the requests coming to NiFi 
are pure HTTP or do I have to proxy the TLS traffic directly to NiFi and 
configure the keystore and truststore on it?


Best regards,
Damian

RE: [NIFI-CPP] Issues with TIMER_DRIVEN and CRON_DRIVEN

2018-04-23 Thread Mendez Veiga, Iyan 
Ok, thanks Marc!

And do they work for you both TIMER_DRIVEN and EVENT_DRIVEN in the raspberry? 
Can you send me a config.yml you have using the GetUSBCamera?

Best regards,
Iyán


De: Marc [mailto:phroc...@apache.org]
Enviado el: lunes, 23 de abril de 2018 0:22
Para: users@nifi.apache.org
Asunto: Re: [NIFI-CPP] Issues with TIMER_DRIVEN and CRON_DRIVEN

**This Message originated from a Non-ArcelorMittal source**
I wanted to follow up and say that timer driven and event driven are the only 
scheduling strategies available in minifi c++. Cron is not a supported strategy 
yet. Sorry for not mentioning this earlier

On Sun, Apr 22, 2018, 4:05 PM Iyán Méndez Veiga 
> wrote:
Hi,

I haven't been able to configure MiNiFi to take pictures every 30 seconds
using GetUSBCamera processor. I have the following in the config.yml

  - name: take_picture
class: GetUSBCamera
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 30 sec
auto-terminated relationships list:
  - failure
Properties:
  FPS: 1
  Format: PNG
  USB Vendor ID: 0x046d
  USB Product ID: 0x082d

The processor ignores the scheduling period and it takes pictures
continuously. I tried to use the CRON_DRIVEN but any processor seems to work.
Perhaps is that I didn't understand how to use it reading this:
http://www.quartz-scheduler.org/documentation/quartz-2.x/tutorials/
crontrigger.html

  - name: take_picture
class: GetUSBCamera
max concurrent tasks: 1
scheduling strategy: CRON_DRIVEN
scheduling period: 0/30 * * * * *
auto-terminated relationships list:
  - failure
Properties:
  FPS: 1
  Format: PNG
  USB Vendor ID: 0x046d
  USB Product ID: 0x082d

Hope anyone can help me. By the way, is there any place to check current open/
known issues and bugs?

Thanks,
Iyán

--
Iyán Méndez Veiga | Physicist
GPG: 0x422E3694311E5AC1
Web: 
https://iyanmv.com

♫♪.ılılıll|̲̅̅●̲̅̅|̲̅̅=̲̅̅|̲̅̅●̲̅̅|llılılı.♫♪

Re: [NIFI-CPP] Issues with TIMER_DRIVEN and CRON_DRIVEN

2018-04-23 Thread Marc P. 
Iyán,

   Yes, both strategies seem to work.

   Here is my config file
   https://gist.github.com/phrocker/a209b5163044a5bb3c2e6ac4662ce143

   Below is the output form my pi. Due to the time it takes to create
a PNG and the scheduling strategy it usually ends up being about a
second between pictures. When I reduce my configuration to what you
have ( for the timing config options ) I maintain the same pattern of
taking a picture every 900 ms. Logs may help further diagnose the
issue.

[2018-04-23 11:11:17.484]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:18.690]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:19.850]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:20.998]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:22.145]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:23.284]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:24.427]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:25.573]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:26.718]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:27.860]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:28.998]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:30.143]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:31.286]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:32.429]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:33.573]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:34.717]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:35.875]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:36.999]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:38.129]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:39.255]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:40.384]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:41.513]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame
[2018-04-23 11:11:42.639]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got
frame

On Mon, Apr 23, 2018 at 6:27 AM, Mendez Veiga, Iyan
 wrote:
> Ok, thanks Marc!
>
>
>
> And do they work for you both TIMER_DRIVEN and EVENT_DRIVEN in the
> raspberry? Can you send me a config.yml you have using the GetUSBCamera?
>
>
>
> Best regards,
>
> Iyán
>
>
>
>
>
> De: Marc [mailto:phroc...@apache.org]
> Enviado el: lunes, 23 de abril de 2018 0:22
> Para: users@nifi.apache.org
> Asunto: Re: [NIFI-CPP] Issues with TIMER_DRIVEN and CRON_DRIVEN
>
>
>
> **This Message originated from a Non-ArcelorMittal source**
>
> I wanted to follow up and say that timer driven and event driven are the
> only scheduling strategies available in minifi c++. Cron is not a supported
> strategy yet. Sorry for not mentioning this earlier
>
>
>
> On Sun, Apr 22, 2018, 4:05 PM Iyán Méndez Veiga  wrote:
>
> Hi,
>
> I haven't been able to configure MiNiFi to take pictures every 30 seconds
> using GetUSBCamera processor. I have the following in the config.yml
>
>   - name: take_picture
> class: GetUSBCamera
> max concurrent tasks: 1
> scheduling strategy: TIMER_DRIVEN
> scheduling period: 30 sec
> auto-terminated relationships list:
>   - failure
> Properties:
>   FPS: 1
>   Format: PNG
>   USB Vendor ID: 0x046d
>   USB Product ID: 0x082d
>
> The processor ignores the scheduling period and it takes pictures
> continuously. I tried to use the CRON_DRIVEN but any processor seems to
> work.
> Perhaps is that I didn't understand how to use it reading this:
> http://www.quartz-scheduler.org/documentation/quartz-2.x/tutorials/
> crontrigger.html
>
>   - name: take_picture
> class: GetUSBCamera
> max concurrent tasks: 1
> scheduling strategy: CRON_DRIVEN
> scheduling period: 0/30 * * * * *
> auto-terminated relationships list:
>   - failure
> Properties:
>   FPS: 1
>   Format: PNG
>   USB Vendor ID: 0x046d
>   USB Product ID: 0x082d
>
> Hope anyone can help me. By the way, is there any place to check current
> open/
> known issues and bugs?
>
> Thanks,
> Iyán
>
> --
> Iyán Méndez Veiga | Physicist
> GPG: 0x422E3694311E5AC1
> Web: https://iyanmv.com
>
> ♫♪.ılılıll|̲̅̅●̲̅̅|̲̅̅=̲̅̅|̲̅̅●̲̅̅|llılılı.♫♪

Generate a surrogate key/sequence for each avro record

2018-04-23 Thread Mohit 
Hi, 

 

I'm trying to added some extra info like a surrogate key/sequence number to
the data fetch from rdbms. 

Is there a way to generate surrogate key for each avro record in nifi?

 

Regards,
Mohit

RE: [NIFI-CPP] Issues with TIMER_DRIVEN and CRON_DRIVEN

2018-04-23 Thread Andy Christianson 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Iyán,

The scheduling for GetUSBCamera is a bit different. The docs (PROCESSORS.md)
say it "outputs one frame per flow file" --it should probably say "one flow
file per frame."

The reason this processor is a bit different is that the underlying camera
library (libuvc) registers a callback that we provide, which is called back at
the given FPS. In this callback, we generate and transfer a new flow file.

This is done so that the flow file outputs are synchronized with the frame rate
of the camera. This reduces latency and allows us to avoid needing a temporary
buffer for onTrigger to copy from. The downside is that the configurating is
different (scheduling is affected only by FPS, nothing else).

The onTrigger function of GetUSBCamera does nothing, so the usual scheduling
mechanisms will do nothing.

As for the image dimensions/quality, GetUSBCamera will enumerate all supported
modes for the camera and select the highest dimensions/quality for the given
FPS. The dimensions are dictated by the camera hardware/drivers, and not us. In
order to change the dimensions after collecting frame data, an additional
expensive resize operation would need to be performed, so it may be better to
have that explicitly done in a follow-on processor. This is something we could
look at changing in GetUSBCamera, but we should carefully consider the
efficiency and configuration UX. A possibility might be to let the user specify
a preferred format in the config, and select the closest supported by the
hardware/drivers. We could also look at supporting resize in the processor, but
that has some downsides (mainly efficiency).

If you are using the images as inputs to a TensorFlow graph, a good place to
resize may be at the start of the TensorFlow graph itself. That way the resize
operation can be parallelized and possibly hardware accelerated.

I have added a ticket [1] to clarify this in the documentation.

Regards,

Andy I.C.

[1] https://issues.apache.org/jira/browse/MINIFICPP-464
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJa3fU6AAoJEG1+mBKNMpID16cIAK7Tk3ykSzCE/9TLb2dLvOyf
jWd0Ced/KhjKBBA+eN6iHxe4eYs/gQtP+Rix+kP93gLTPrZ7vF4Cxb3jT3zra6pI
DaCxp6zUzcGtd960XXnQ1wEii1rIyamokuK8A4uIUFKgcZK6WKgRlcGr6lOWQ5c3
J6/VJNb/eL076lDq6Xj4zU0msxsTMUZWBXJjZ5K6JlI+Y983u+bfNKqnUArWgt/z
KIp8DIBXnib7xNQAIrk8k7THZkS+0/SENSt9j8OV/P2rkXUaWmBwFSX1F4s+k9Dq
4iUvga9mSwvuThR7IiJqrcDJjwo09qHWOVAoTPkzcjYlIGGshUOeIoHcmWKJV24=
=3esw
-END PGP SIGNATURE-



​Sent from ProtonMail, Swiss-based encrypted email.​

‐‐‐ Original Message ‐‐‐

On April 23, 2018 10:35 AM, Mendez Veiga, Iyan 
 wrote:

> ​​
> 
> Ok, thanks Marc. I have tried your file (just making it simpler and adding a 
> RPG) and timer does not work for me... I will send the log later when I 
> arrive home.
> 
> So, just a silly question. Imagine I want a picture every 10 seconds. I would 
> just have to change scheduling period to 10 sec or 1 msec, right?. Does 
> this work even if I specify FPS: 24?
> 
> I noticed that the video profile is chosen based on the FPS value. If I set 
> FPS: 1 I get very large images while with FPS: 24 I get smaller 800x600 
> images. I guess this will be configured as a property in the future.
> 
> Best regards,
> 
> Iyán
> 
> -Mensaje original-
> 
> De: Marc P. [mailto:marc.par...@gmail.com]
> 
> Enviado el: lunes, 23 de abril de 2018 13:16
> 
> Para: users@nifi.apache.org
> 
> Asunto: Re: [NIFI-CPP] Issues with TIMER_DRIVEN and CRON_DRIVEN
> 
> This Message originated from a Non-ArcelorMittal source
> 
> Iyán,
> 
> Yes, both strategies seem to work.
> 
> Here is my config file
> 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_phrocker_a209b5163044a5bb3c2e6ac4662ce143=DwIFaQ=y5LGzd1hT50ruE_IlUH7x8VGgWz9W0tFVWT6rSvPUKA=OqJOXvZGIFafgLt7Uocjlzsiwprj2_HJJ0p7HG9IsupasVjguMhbhoUfzdA5V_Qn=CCOIqqrapqve7SfBS7uoDgFCrVh83PSU7mN-ZblHnho=ehLOY9yBJ2LpM0Jnl-xWpiL5r35oVQnvoglF9U_m118=
> 
> Below is the output form my pi. Due to the time it takes to create a PNG and 
> the scheduling strategy it usually ends up being about a second between 
> pictures. When I reduce my configuration to what you have ( for the timing 
> config options ) I maintain the same pattern of taking a picture every 900 
> ms. Logs may help further diagnose the issue.
> 
> [2018-04-23 11:11:17.484]
> 
> [org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
> 
> [2018-04-23 11:11:18.690]
> 
> [org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
> 
> [2018-04-23 11:11:19.850]
> 
> [org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
> 
> [2018-04-23 11:11:20.998]
> 
> [org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
> 
> [2018-04-23 11:11:22.145]
> 
> [org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
> 
> [2018-04-23 11:11:23.284]
> 
>

RE: [NIFI-CPP] Issues with TIMER_DRIVEN and CRON_DRIVEN

2018-04-23 Thread Mendez Veiga, Iyan 
Ok, thanks Marc. I have tried your file (just making it simpler and adding a 
RPG) and timer does not work for me... I will send the log later when I arrive 
home.
So, just a silly question. Imagine I want a picture every 10 seconds. I would 
just have to change scheduling period to 10 sec or 1 msec, right?. Does 
this work even if I specify FPS: 24?

I noticed that the video profile is chosen based on the FPS value. If I set 
FPS: 1 I get very large images while with FPS: 24 I get smaller 800x600 images. 
I guess this will be configured as a property in the future.

Best regards,
Iyán

-Mensaje original-
De: Marc P. [mailto:marc.par...@gmail.com] 
Enviado el: lunes, 23 de abril de 2018 13:16
Para: users@nifi.apache.org
Asunto: Re: [NIFI-CPP] Issues with TIMER_DRIVEN and CRON_DRIVEN

**This Message originated from a Non-ArcelorMittal source**


Iyán,

   Yes, both strategies seem to work.

   Here is my config file
   
https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_phrocker_a209b5163044a5bb3c2e6ac4662ce143=DwIFaQ=y5LGzd1hT50ruE_IlUH7x8VGgWz9W0tFVWT6rSvPUKA=OqJOXvZGIFafgLt7Uocjlzsiwprj2_HJJ0p7HG9IsupasVjguMhbhoUfzdA5V_Qn=CCOIqqrapqve7SfBS7uoDgFCrVh83PSU7mN-ZblHnho=ehLOY9yBJ2LpM0Jnl-xWpiL5r35oVQnvoglF9U_m118=

   Below is the output form my pi. Due to the time it takes to create a PNG and 
the scheduling strategy it usually ends up being about a second between 
pictures. When I reduce my configuration to what you have ( for the timing 
config options ) I maintain the same pattern of taking a picture every 900 ms. 
Logs may help further diagnose the issue.

[2018-04-23 11:11:17.484]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:18.690]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:19.850]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:20.998]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:22.145]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:23.284]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:24.427]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:25.573]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:26.718]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:27.860]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:28.998]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:30.143]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:31.286]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:32.429]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:33.573]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:34.717]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:35.875]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:36.999]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:38.129]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:39.255]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:40.384]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:41.513]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame
[2018-04-23 11:11:42.639]
[org::apache::nifi::minifi::processors::GetUSBCamera] [debug] Got frame

On Mon, Apr 23, 2018 at 6:27 AM, Mendez Veiga, Iyan 
 wrote:
> Ok, thanks Marc!
>
>
>
> And do they work for you both TIMER_DRIVEN and EVENT_DRIVEN in the 
> raspberry? Can you send me a config.yml you have using the GetUSBCamera?
>
>
>
> Best regards,
>
> Iyán
>
>
>
>
>
> De: Marc [mailto:phroc...@apache.org]
> Enviado el: lunes, 23 de abril de 2018 0:22
> Para: users@nifi.apache.org
> Asunto: Re: [NIFI-CPP] Issues with TIMER_DRIVEN and CRON_DRIVEN
>
>
>
> **This Message originated from a Non-ArcelorMittal source**
>
> I wanted to follow up and say that timer driven and event driven are 
> the only scheduling strategies available in minifi c++. Cron is not a 
> supported strategy yet. Sorry for not mentioning this earlier
>
>
>
> On Sun, Apr 22, 2018, 4:05 PM Iyán Méndez Veiga  wrote:
>
> Hi,
>
> I haven't been able to configure MiNiFi to take pictures every 30 
> seconds using GetUSBCamera processor. I have the following in the 
> config.yml
>
>   - name: take_picture
> class: GetUSBCamera
> max concurrent tasks: 1
> scheduling strategy:

Best way to drop data based on its content

2018-04-23 Thread Laurens Vets 

Hello list,

I'm using NiFi to ship JSON formatted data around. However, I want NiFi 
to drop certain data when some field matches. Let's say we have a field 
called 'username' and we don't want to further process any data which 
contains "username":"laurens".


What I currently do is get the field 'username' out of the JSON and add 
it to the flowfile attribute and then drop when the flowfile matches. I 
use RouteOnAttribute for this and automatically terminate the 
relationship when it matches.


Is there a better/faster way? Thanks in advance!

Re: Best way to drop data based on its content

2018-04-23 Thread James Wing 
Your solution sounds very normal and appropriate to me.  Is it performing
slowly or causing you problems?

Thanks,

James

On Mon, Apr 23, 2018 at 2:37 PM, Laurens Vets  wrote:

> Hello list,
>
> I'm using NiFi to ship JSON formatted data around. However, I want NiFi to
> drop certain data when some field matches. Let's say we have a field called
> 'username' and we don't want to further process any data which contains
> "username":"laurens".
>
> What I currently do is get the field 'username' out of the JSON and add it
> to the flowfile attribute and then drop when the flowfile matches. I use
> RouteOnAttribute for this and automatically terminate the relationship when
> it matches.
>
> Is there a better/faster way? Thanks in advance!
>

RE: Generate a surrogate key/sequence for each avro record

2018-04-23 Thread Mohit 
I'm able to achieve it using QueryRecord processor.

 

Thanks,

Mohit

 

From: Mohit  
Sent: 23 April 2018 15:59
To: users@nifi.apache.org
Subject: Generate a surrogate key/sequence for each avro record

 

Hi, 

 

I'm trying to added some extra info like a surrogate key/sequence number to
the data fetch from rdbms. 

Is there a way to generate surrogate key for each avro record in nifi?

 

Regards,
Mohit