Re: upstream system is invalid

2016-10-07 Thread Aldrin Piri
Hey Niraj, What the error is covering is that this processor requires input and does not act as a source processor. Instead, this processor will perform operations based on FlowFile input and will make use of incoming FlowFile's attributes in order to select items from DynamoDB. Said attributes

Re: Build failure under CentOS 6.7

2016-10-18 Thread Aldrin Piri
Hey Michael, Can you confirm the JDK is installed and available from your build? Does `javac -version` return appropriately? Is JAVA_HOME set? If not, could you set it to the location of your JDK? If it is, can you verify the bin directory associated with it contains a javac? The culprit seem

Re: attributesToJSON order

2016-10-20 Thread Aldrin Piri
Hey Paul, Could you highlight the use case you are looking to address or shortcoming that has emerged because of this? No strong qualms with providing it, just not sure I am tracking where this becomes problematic. Thanks, Aldrin On Thu, Oct 20, 2016 at 3:23 PM, Paul Gibeault (pagibeault) < pag

Re: SplitJson:GC Overhead Limit Exceeded

2016-11-17 Thread Aldrin Piri
The backing library of the Json processors does indeed require loading the entire doc into memory. We should make sure this consideration is documented if not already. Could be an interesting idea to not tie SplitJson to this library given that it might not need all the functionalities of JsonPath

Re: Processors and

2016-11-30 Thread Aldrin Piri
Hi Andreas, 1) There is nothing from a framework perspective that provides this. However, a typical option is to make use of an attribute from an upstream processor to help categorize and handle the data. Attributes written vary from processor to processor or can be explicitly set/updated using

Re: Apache Minifi questions

2016-12-14 Thread Aldrin Piri
Juan, MiNiFi should be fine with 0.7.x in terms of generating a template to be converted for usage in MiNiFi. Sending data to a NiFi instance also works across site to site to the same version. Of course other transport mechanisms across differing protocols are also fair game. Currently, proven

Re: Process Group singleton

2016-12-27 Thread Aldrin Piri
Hi Yari, The concept you are looking for has been discussed a bit and a draft feature proposal was created for referenceable process groups [1], although is a common pattern and implementable with the current functionality provided; common cases of this are standard tagging or enrichment sets of f

Re: Process Group singleton

2016-12-27 Thread Aldrin Piri
hy I was looking for something that could be referenced from anywhere > (like a RPG). > > Yari > > On 27 December 2016 at 16:14, Aldrin Piri wrote: > >> Hi Yari, >> >> The concept you are looking for has been discussed a bit and a draft >> feature proposal wa

Re: Apache nifi FLow.xml guidance

2017-01-23 Thread Aldrin Piri
dropping dev list Hi Matt, Could you provide additional details and/or a flow of what you have attempted to do thus far? Are you looking to filter out .doc files but allow images? On Mon, Jan 23, 2017 at 4:29 AM, mtn jira wrote: > Hi All, > > I am new to Apche nifi and we have a setup where we

Re: Custom processors in MiNiFi

2017-01-27 Thread Aldrin Piri
Hi, Aakash! To my knowledge, I have not seen any discussion about such processors on the lists specifically although have heard people mentioning assorted libraries that might be a good fit for the NiFi ecosystem's intended purposes. There has been some foundational work such as the following iss

Re: Minifi question

2017-01-30 Thread Aldrin Piri
Hi Juan, The FlowStatus mechanism aids in inquiring about the current state of a running instance. Additional details about this can be found in the admin guide: https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/markdown/System_Admin_Guide.md#status-reporting-and-querying Pr

Re: GetTwitter - Security/Certificate Issue

2017-01-31 Thread Aldrin Piri
Hi Dan, The GetTwitter processor does not make use of an Apache NiFi SSLContextService so the certificate chain issues are likely more tied to the JVM/OS specifically. Did a quick check on some of the instances I am running and Twitter seems to be operating normally. Could you share some more de

Re: GetTwitter - Security/Certificate Issue

2017-02-01 Thread Aldrin Piri
info to help diagnose the issue! > > > > Thanks, > > > > Dan > > > > > > *From:* Aldrin Piri [mailto:aldrinp...@gmail.com] > *Sent:* Tuesday, January 31, 2017 2:20 PM > *To:* users@nifi.apache.org > *Subject:* Re: GetTwitter - Security/Certificate

Re: how to import a template

2017-02-06 Thread Aldrin Piri
Hello! This is located on the "Operate" palette. The User Guide has some additional details on it: http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Import_Template On Mon, Feb 6, 2017 at 8:43 PM, lk_hadoop wrote: > I did't find import button on NiFi Templates page. I am using nifi

Re: GetTwitter - Security/Certificate Issue

2017-02-06 Thread Aldrin Piri
ur ‘sandbox’ to > try new things (nifi in this case), so I can definitely upgrade the yum > package. As for your second question, the server runs on my company’s > network, but other than that I don’t see any other considerations. Any > thoughts? > > > > -Dan > > > &g

Re: How to send back a reply from HandleHTTPRequest

2017-02-07 Thread Aldrin Piri
HI James, This would occur via the HandleHTTPResponse. At a high level, the request flowfile is routed to the HandleHTTPResponse processor. Of course, any kind of processing could occur between the two points. A simple example can be located on the sample templates page with the Hello_NiFi_Web_

Re: GetTwitter - Security/Certificate Issue

2017-02-07 Thread Aldrin Piri
Tue, Feb 7, 2017 at 4:39 PM, Dan Giannone wrote: > Hi Aldrin, > > > > Here is a screenshot of the result. Looks like there is definitely an > issue. Please let me know if this sheds any light on the issue. > > > > Thanks, > > > > Dan > > > > *Fr

Re: NiFi Users: Powered by NiFi page

2017-02-15 Thread Aldrin Piri
Hi Giovanni, GoDataDriven has been included. Thanks! On Wed, Feb 15, 2017 at 11:48 AM, Giovanni Lanzani < giovannilanz...@godatadriven.com> wrote: > Hi Joe, > > You can put GoDataDriven (https://godatadriven.com) > Summary: GoDataDriven, a Dutch service company in the data science and > enginee

Re: Cannot Restart Nifi

2017-03-28 Thread Aldrin Piri
wfile Repo should get you going. You can additionally remove the content repo, but this should be cleaned up by the framework as no flowfiles will point to said content. Aldrin Piri Sent from my mobile device. > On Mar 28, 2017, at 06:12, James McMahon wrote: > > I noticed, too, th

Re: Cannot Restart Nifi

2017-03-28 Thread Aldrin Piri
o come back up. > > Please tell me: while I am in a dev state right now, had I been in a > production state what would have been the repercussions of deleting in its > entirety the flowfile_repository, which includes all its journal files? > > Thanks very much in advance for your help.

Re: Cannot Restart Nifi

2017-03-28 Thread Aldrin Piri
empt to start NiFi > > 4- if 3 fails, rm -rf ./flowfile_repository on this, my dev box. Start > nifi, > > get in there, and eliminate that Python logging. Find another way to log > > results to a system file, perhaps using a NiFi processor. > > > > - Jim > > >

Re: Connection cannot be established

2017-03-28 Thread Aldrin Piri
Hi Otmane! Dropping dev to BCC. We do not currently have support for this in the processor as currently released and is an outstanding JIRA issue [1]. On that issue, a community member has an implementation that may work for you but also seems dependent on the core library of the processor (hbc)

Re: How does lib_community get included in NiFi nar search path?

2017-03-29 Thread Aldrin Piri
Hi Jim, lib_community is not one of the default directories NiFi searches. You can update nifi.properties to include additional paths via properties such as those outlined in the System Administrator's guide in Core Properties for nifi.nar.library.directory [1]. The description of this follows:

Re: RouteOnContent--how to specify the promised dynamic relationship?

2017-03-30 Thread Aldrin Piri
Russell, Your expectations are inline with the functionality of the processor. You can add dynamic relationship via a dynamic property in the processor properties tab. The plus (+) located at the top right will let you first specify a property name (Treated as the relationship) and its value wou

Re: Preserve or replace 'state' directory during upgrade - NiFi 1.1.2

2017-03-30 Thread Aldrin Piri
Mark beat me to the reply but it does appear our documentation is lacking. I've opened up an issue to include/improve this aspect in one of our upgrade and/or admin guides. https://issues.apache.org/jira/browse/NIFI-3665 Good question and thanks for reaching out. On Thu, Mar 30, 2017 at 7:31 PM,

Re: Configure ExecuteSQL in MiNiFi

2017-04-13 Thread Aldrin Piri
Hi Jeff, This support was provided and is slated for 0.2.0 via MINIFI-154 but has not yet been released pending NiFi 1.2.0. From your question you had on another issue yesterday, I believe you are building from master so you should have the requisite functionality needed. For SQL work, relevant

Re: Netflow parser

2017-04-17 Thread Aldrin Piri
Hi Corey, Does not look like there has been an implementation at this point in time. There is a JIRA that is tracking the request of this thread: https://issues.apache.org/jira/browse/NIFI-2904 On Mon, Apr 17, 2017 at 10:57 AM, Corey Flowers wrote: > Good morning everyone, > > Was there e

Re: MiNiFi's differentiator

2017-04-20 Thread Aldrin Piri
Hey Jeff, Just to hone in a bit more, MiNiFi would likely consume from some of the work surrounding the Command and Control [1] efforts. There is some initial work under way which should serve as a nice foundation to start exploring a more full featured and richer implementation. Certainly plan

Re: Security between MiNiFi and NiFi

2017-04-21 Thread Aldrin Piri
Hi Varsha, The preferred mechanism for accomplishing this is via Site to Site and Remote Processing Groups [1]. This makes use of the configured Security Properties [2]. For other means of data egress, these would vary depending on specific processor or extension used. [1] https://github.com/ap

Re: Connecting to OPCUA

2017-04-21 Thread Aldrin Piri
Hi Varsha, Went through the build process quickly on my side building the dependent UA Java and then referencing that version in the listed project with some cleanup and had the GetNodeIds processor available in NiFi after copying the resulting NARs into my lib directory. Doing some looking throu

Re: How to identify MiNiFi source edge devices

2017-04-25 Thread Aldrin Piri
Jeff, Could you expand upon what a device id is in your case? Something intrinsic to the device? The agent? Are these generated and assigned during provisioning? How are you making use of these when the data arrives at its desired destination? What you are expressing is certainly a common nee

Re: Configure duration to push to RPG in AWS (IoT flow)

2017-05-01 Thread Aldrin Piri
Hi Varsha, When you are talking about flows, are these separate instances of MiNiFi or different parts of your overall configuration? Is there a reason the collected data is written to and then read from disk? I/O costs are amplified when running off of SD cards which can be quite slow. Not tha

Re: [ANNOUNCE] Apache NiFi 1.2.0 release

2017-05-10 Thread Aldrin Piri
Hey Juan, We had a small issue with the auto-build process. Will be working with INFRA to get that generated and hope to have resolved shortly. On Wed, May 10, 2017 at 11:19 AM, Juan Pablo Gardella < gardellajuanpa...@gmail.com> wrote: > The release notes mention an official image docker. Do yo

Re: [ANNOUNCE] Apache NiFi 1.2.0 release

2017-05-10 Thread Aldrin Piri
Juan, Just wanted to follow up that INFRA was most helpful and able to trigger our build for us. The associated Hub repo is available at https://hub.docker.com/r/ apache/nifi/ docker run -P apache/nifi:1.2.0 will pull/run an instance On Wed, May 10, 2017 at 11:26 AM, Aldrin Piri wrote: >

ASF NiFi HipChat room now available

2017-05-10 Thread Aldrin Piri
Hey folks, Thanks to the ASF Infra team we have a public HipChat room available. The hope for this is to serve as another avenue for communication as well as facilitating discussion for cases such as those when something more direct than GitHub comments is appropriate. The room is reachable via

Re: some processors runs only once in NiFi

2017-05-23 Thread Aldrin Piri
For GetHTTP, this processor makes use of ETags[1] to prevent downloading the same resource repeatedly. I would speculate that this is the case for the resource you are specifying. As for ListHDFS, could you specify what version you are using? There have been some bugs concerning how this was han

Re: NiFi will not restart, missing status file message in bootstrap

2017-05-25 Thread Aldrin Piri
If you happen to remember, could you get more specific into your sequence of operations? Is nifi installed as a service? If so, was it restarted Did you just issue a nifi.sh restart? Do you have any CM tooling (Puppet, Chef, Salt, etc) that is managing this process/system? Could you tell us what

Re: Unable to start NiFi on an EC2 instance

2017-06-12 Thread Aldrin Piri
Hi James! NiFi 1.0+ explicitly requires Java 8. As far as service installation, it is likely needed to be done as root for touching those parts of the system. Would also suggest using the script to install the service. Let us know if you hit hurdles after getting JRE 8 installed. Aldrin

Re: MiNiFi on Ubuntu Core

2017-07-17 Thread Aldrin Piri
Hi Pierre & Gabor, There is no work to this that I know of nor can I find any issues tracking this in JIRA. Gabor, if you would be willing to contribute to the effort, I would encourage you to create a JIRA so there is a central place to track and discuss the effort. Thanks! --aldrin On Mon, Ju

Re: MaxHistoryNot Worked in logback-1.2.3?

2017-07-18 Thread Aldrin Piri
I think the comment in the config XML may be a bit misleading. Looking through the docs, it appears your configuration would keep 2 days of log files given your filename format leading to daily logs. Your max size of 100kB will cause rollover, incrementing the value represented by %i. With this

Re: Using QueryDatabaseTable processor in MiNiFi

2017-07-27 Thread Aldrin Piri
Hey Jeff, Could you please provide a full nifi-app.log of the startup? Thanks, Aldrin On Thu, Jul 27, 2017 at 10:04 AM, Jeff Zemerick wrote: > I have a small flow that was exported from NiFi and it uses a > QueryDatabaseTable processor. When the flow is converted to yaml and set in > MiNiFi I

Re: Using QueryDatabaseTable processor in MiNiFi

2017-07-27 Thread Aldrin Piri
was total user > failure somewhere on my part. I am curious as to where I went wrong so I am > trying to reproduce it to see if it's worth any documentation updates. > > Thanks, > Jeff > > > On Thu, Jul 27, 2017 at 10:09 AM, Aldrin Piri > wrote: > >> Hey

Re: NiFi Docker

2017-07-31 Thread Aldrin Piri
Hi Dave, Yes! Most certainly to all these things you are suggesting. Would you be interested in opening a JIRA to capture these? If not, I will be sure to do so. Definitely has been an item to address moving forward as you are a limited with the current image. We may not be able to support eve

Re: NiFi Docker

2017-08-01 Thread Aldrin Piri
Sounds great. Will keep an eye out for it. Thanks! On Tue, Aug 1, 2017 at 10:50 AM, ddewaele wrote: > Great also have some ideas about this. I'll log a JIRA and elaborate > on > those. > > We can then see on how to move this forward. (willing to do a pull request > for this). > > > > -- >

Re: Control and Command information between MiniFi and NiFi

2017-08-16 Thread Aldrin Piri
Hi Vikram, Good questions and thought out process in this email. Your approach seems valid and a fairly typical use case which we are working on supporting well. 1) At this time, you cannot do so directly from NiFi. I think based on some of the initial discussions and work that has been perfor

Re: List of processors supported by MiNiFi 0.2.0

2017-08-17 Thread Aldrin Piri
Hi Jeremy, Thanks for bringing the lack of a listing to our attention. We actually have documentation created to support this [1] but failed to actually publish and/or reference it anywhere. We address that with MINIFI-379 [2]. To the issue at hand, it is possible to make use of NARs from NiFi

Re: MinFi Site2Site

2017-08-23 Thread Aldrin Piri
Hi Vikram, It is not possible at the moment to do so given that the Site to Site protocol makes use of the nifi-api as a server to communicate endpoints for a client to transmit to. That certainly doesn't preclude us from getting to this point in the future but will require some consideration on

Re: Using invokeHTTP with binary compression enabled endpoints

2017-08-23 Thread Aldrin Piri
Hi Raman, InvokeHTTP can send data of any format. The issue at hand is that specifying a content type does not performa any transformation of the source flowfile content to transmit. Rather, this is just establishing the Content-Type header. To that end, what you are seeking is certainly possi

Re: Using MiNiFi as a library from with a Java program

2017-09-26 Thread Aldrin Piri
Hello, Neither NiFi nor MiNiFi (Java) is currently packaged as such. Would you be able to expand upon your use case of publishing to Kafka and/or what facets of MiNiFi you are looking to utilize? There may be other avenues that get you toward your solution. Thanks, Aldrin On Tue, Sep 26, 2017

Re: MiniFi

2017-10-05 Thread Aldrin Piri
Hi Vikram, This is not currently possible. There has been some initial work and some discussion on C2 capabilities that are in progress of being designed/built out [1][2]. There is heartbeating and providing operations for an agent to carry out that has an initial implementation in C++. There i

Re: Capture Change Mysql Processor is unable to start using minifi

2017-10-13 Thread Aldrin Piri
The library in question is, in fact one that was troublesome and removed in MINIFI-403. Could you please share what your lib directory looks like and please verify the hash you built from for 0.2.1? I revisited the work I did for 403 and am able to use the flow provided there that is also making

Re: Capture Change Mysql Processor is unable to start using minifi

2017-10-13 Thread Aldrin Piri
Hi Rakesh, The nifi-distributed-cache-client-service-api.jar is the issue. Building off latest master, that does not appear in the lib directory for me. Perhaps that was accidentally copied? The nifi-standard-services-api-nar contains the aforementioned jar. As a general point of consideration,

Re: Capture Change Mysql Processor is unable to start using minifi

2017-10-16 Thread Aldrin Piri
Looks as though the cache was never started and it was then not possible to connect to it. Are there other errors when the flow initializes concerning the cache's initialization? On Mon, Oct 16, 2017 at 8:23 AM, rakesh wrote: > Hi Aldrin, > > Thanks for your help. The error reported is goneaway

Re: Capture Change Mysql Processor is unable to start using minifi

2017-10-16 Thread Aldrin Piri
Do you also have a DistributedMapCacheServer controller service configured in your config? I do not see one of these instantiated in your log. This would explain why the client service is getting the connection refused events. On Mon, Oct 16, 2017 at 9:53 AM, rakesh wrote: > Exception started

Re: Capture Change Mysql Processor is unable to start using minifi

2017-10-16 Thread Aldrin Piri
Unfortunately templates exported from NiFi do not contain Controller Services that are not referenced by one of the selected extensions. As a result, yes, this would be another Controller Services entry. Something to the effect of the following should work: - id: name: Map Cacher Server type:

Re: MiNiFi & Certs/Keys

2017-10-20 Thread Aldrin Piri
I am a far cry from a cert pro, but it looks like the results are what I would anticipate given your reuse of nifi_server.key.pem in your JKS. I don't think you are far away from your intended setup and should be able to create another "user" cert for your minifi instance(s), as you had for your U

Re: minifi release and mapping that to the nifi release

2017-11-21 Thread Aldrin Piri
Hi Juan, We need to have this documented in terms of the core NiFi library versions used in MiNiFi Java. The general contract is that we can support extensions in the same manner that the framework libraries of NiFi does. This had a few hurdles which will be resolved with MINIFI-408 [1]. I'll cr

Re: HandleHttpRequest

2018-01-03 Thread Aldrin Piri
Are you receiving an incoming FlowFile in the HandleHttpRequest? Could you share a template? Alternatively, as a bit of inspiration, you could also evaluate the Hello_NiFi_Web_Service.xml [1] from https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates to see if there is anyth

Re: HandleHttpRequest

2018-01-03 Thread Aldrin Piri
generated. a template of what >> >> On Wed, Jan 3, 2018 at 4:24 PM, Aldrin Piri wrote: >> >>> Are you receiving an incoming FlowFile in the HandleHttpRequest? >>> >>> Could you share a template? >>> >>> Alternatively, as a bit of inspirati

Re: MiNiFi contents-repository config problem

2018-01-16 Thread Aldrin Piri
Hi Saulo, Might you be able to specify the version you are using? If possible, could you also share your config.yml? The second point is expected behavior as the config file is transformed into nifi.properties. However, the items you are trying to adjust are not enabled as archiving is disabled

Re: MiNiFi contents-repository config problem

2018-01-16 Thread Aldrin Piri
Hi Saulo, It looks like the config unfortunately got stripped from the upload. Would you mind opening an issue at https://issues.apache.org/jira/projects/MINIFI/issues and attaching it there please? On Tue, Jan 16, 2018 at 9:50 AM, Saulo wrote: > Hi Aldrin, > > > /Might you be able to specif

Re: MiNiFi contents-repository config problem

2018-01-16 Thread Aldrin Piri
Great, thanks! Will put that on my list to scope out and will move conversation over to the issue. On Tue, Jan 16, 2018 at 10:32 AM, Saulo wrote: > Hi Aldrin, > > > Sure, I have just created the issue with the information we shared and > attached the file. > > Issue link: https://issues.apache.o

Re: Output Port Routing and DRYing up Nifi processor Groups

2015-08-10 Thread Aldrin Piri
John, In short, not at the moment, but there are plans to facilitate this process much in the way as you described. I think you would be greatly aided by a concept that Joe presented for discussion to the community. This is only in design, but there are many people that are mapping components of

Re: New to NiFi - Remote Process Group failing to connect due to "magic header" not present

2015-08-15 Thread Aldrin Piri
Rick, Welcome to the community! We seem to be a little short in the documentation department for how to make use of Remote Process Groups, but will look to remedy that. Just to confirm a few settings, both your nodes have a nifi.remote.input.socket.port set and each has nifi.remote.input.secure

Re: New to NiFi - Remote Process Group failing to connect due to "magic header" not present

2015-08-15 Thread Aldrin Piri
UI address, > but the separate site-to-site listener, which I see via netstat on target > node) > > > > Rick > > > > *From:* Aldrin Piri [mailto:aldrinp...@gmail.com] > *Sent:* Saturday, August 15, 2015 3:48 PM > *To:* users@nifi.apache.org > *Subject:* Re: Ne

Re: Event-driven process scheduling for RSS feed?

2015-08-23 Thread Aldrin Piri
Rick, You would be correct in your analysis of GetFile. GetFile does not have a processing "mode" where it is driven by incoming files. Just to clarify, when you are referencing scheduling, that is approaching "running" of the processor slightly differently, and dictates when the Flow Controller

Re: Nifi Rest API Examples

2015-09-02 Thread Aldrin Piri
John, The API lives at /nifi-api, so the address you are looking for would be http://localhost:8080/nifi-api/controller/about Please let us know if you have any additional questions or stumbling blocks. Thanks! Aldrin Piri Sent from my mobile device. On Wed, Sep 2, 2015 at 10:16 AM, John

Re: GetHTTP processor to consume public https?

2015-09-04 Thread Aldrin Piri
The issue for those sites is you need a set of root certificates for the common sites on the web much the same way a browser comes bundled with them. The Linux distributions typically come with a prebuilt truststore when Java is installed and it lives within /etc/pki/. Exact location escapes me at

Re: GetHTTP processor to consume public https?

2015-09-05 Thread Aldrin Piri
Sat, 5 Sep 2015 at 2:31 pm Joe Witt wrote: > >> would it be feasible to try and look for reasonable default locations >> to find prebuilt keystores to make this sort of thing easier? >> >> ...in addition to documenting it of course. >> >> Thanks &g

Re: ExtractText usage

2015-09-08 Thread Aldrin Piri
Mark, The need is certainly there. This core functionality is quite similar to what NIFI-921 (sorry for no link, doing mobile) is after. The issue that both of these are addressing is that there isn't a simple way to promote "simple" data to an attribute. There are some important distinctions betw

Re: Array into MongoDB

2015-09-24 Thread Aldrin Piri
Bryan is correct about the backing library reading everything into memory to do the evaluation. Might I ask what the expression you are using? On Thu, Sep 24, 2015 at 6:44 PM, Adam Williams wrote: > I tried it even with 6GB and no luck. It's receiving the flowfiles, but > nothing is happening

Re: Array into MongoDB

2015-09-24 Thread Aldrin Piri
Jeff, This seems to be a bit different as the processor is showing data as having been written and there is a listing of one FlowFile of 381 MB being transferred out from the processor. Could you provide additional information as to how data is not being sent out in the manner anticipated? If yo

Re: NiFi flow provides 0 output on large files

2015-09-25 Thread Aldrin Piri
m processor > shows movement. > > I confirmed that it is not a data problem by processing a 10,000 row file > successfully, then concatenating 10,000 rows into one file twice. > > Thanks for your insight. > > Jeff > > > > On Sep 24, 2015, at 8:40 PM, Aldrin Piri

Re: NiFi flow provides 0 output on large files

2015-09-25 Thread Aldrin Piri
low image shows that the GetFile Processed 174.6 MB and the >>> DDA_Processor is working on 1 file (the 1 in the upper right of the >>> DDA_Processor box) >>> >>> >>> >>> The below image shows that the DDA_Processor is complete but data

Re: Run process after getting a file

2015-09-28 Thread Aldrin Piri
Odin, * Saw Rick's message come in while typing this and I believe this may very well be causing your issue, but don't think changing that will resolve your issues. * Saw your response just come in. This has been a highly interrupted email process spanning a long interval! :) Part of the issue

Re: Run process after getting a file

2015-09-28 Thread Aldrin Piri
GetFtp has the same semantics in what a file is and how it is handled. Once it is received via FTP it will be in the same "state" as if it had been received with a GetFile. On Mon, Sep 28, 2015 at 11:58 AM, Odin Guillermo Caudillo Gallegos < odin.guille...@gmail.com> wrote: > Will be the same if

Re: Run process after getting a file

2015-09-28 Thread Aldrin Piri
the code, > now if i use GetFTP should i use the same Expression Language as input > parameter? > Thanks > > 2015-09-28 10:58 GMT-05:00 Aldrin Piri : > >> Odin, >> >> * Saw Rick's message come in while typing this and I believe this may >> very well b

Re: nifi Cluster setup issue

2015-09-29 Thread Aldrin Piri
Chakrader, I suspect that the nifi.web.http.host property is not using the same address as that specified and is transmitting "localhost" (the system's response to a localhost hostname lookup from Java). While the clustering protocol communicates via the properties you list, the actual command-co

Re: nifi Cluster setup issue

2015-09-29 Thread Aldrin Piri
provider=file-provider > > nifi.security.support.new.account.requests= > > nifi.security.ocsp.responder.url= > > nifi.security.ocsp.responder.certificate= > > > # cluster common properties (cluster manager and nodes must have same > values) # > > nifi.cluster.

Re: nifi Cluster setup issue

2015-09-29 Thread Aldrin Piri
Oops, definitely missed what Corey sent out. Please specify the nifi.cluster.manager.address as he suggests. On Tue, Sep 29, 2015 at 9:40 PM, Aldrin Piri wrote: > Chakrader, > > You would also need to set the nifi.web.http.host for the manager as > well. Each member of the clus

Re: nifi Cluster setup issue

2015-09-30 Thread Aldrin Piri
Chakrader, You certainly can have hybrid nodes although we tend to avoid that if possible. The key point to keep in mind is your allocation of ports for your http server and clustering protocol as there are two processes living on one system. It looks like from the configurations you have previo

Re: Parse JSON issue

2015-10-01 Thread Aldrin Piri
In this case, with your expression, you are saying select all children of the root (objects) and select their "values". This works out to be a nested array and is what you are seeing in the exception. What you are looking for is something more along the lines of: $.[0].values[0] Select the firs

Re: FetchS3Object processor

2015-10-08 Thread Aldrin Piri
Chakri, The FetchS3Object processor does not act as a source directly and is instead driven by incoming FlowFiles. The use case was that one could utilize Expression Language to interact with buckets and paths in a dynamic fashion. Alternatively, if you are looking to just grab a specific file,

Re: Nifi GetHttp https endpoint keystore password error

2015-11-05 Thread Aldrin Piri
Hi Mans, Quite all right and SSL can be a bit overwhelming for those not familiar. We had a user with similar issues as you, please check out the associated thread here: https://mail-archives.apache.org/mod_mbox/nifi-users/201509.mbox/%3ceb8bf379-164b-49d0-82b5-7043f2891...@gmail.com%3E In summa

Re: Nifi GetHttp https endpoint keystore password error

2015-11-05 Thread Aldrin Piri
> > On Thursday, November 5, 2015 3:58 PM, Aldrin Piri > wrote: > > > Hi Mans, > > Quite all right and SSL can be a bit overwhelming for those not familiar. > > We had a user with similar issues as you, please check out the associated > thread here: > https://

Re: Help me figure out why this wont work

2015-11-06 Thread Aldrin Piri
Christopher, For this setup, you want to use GetHTTP. InvokeHTTP operates off of FlowFile input. On Fri, Nov 6, 2015 at 2:33 AM, Christopher Hamm wrote: > I have a dataFlow setup that looks up api call using current date. I turn > it on and nothing seems to happen.. Help appreciated. > > > > -

Re: A "proxy" question from the irc channel

2015-11-08 Thread Aldrin Piri
Mild tangent. I vaguely remember it coming up on the mailing list quite some time ago, but is the IRC channel's information available anywhere on the site? Wasn't able to find anything immediately obvious. On Sun, Nov 8, 2015 at 12:19 PM, Tony Kurc wrote: > and a proxy from irc answer: > > An

Re: Help me figure out why this wont work

2015-11-09 Thread Aldrin Piri
do with > message of "supports expression language: false" for the url. Attached > error and xml template. Any help is appreciated. I need to understand how > the dynamic querying should work in order to complete an actual project. > Many thanks > > > > > > > &g

Re: Reading file

2015-11-09 Thread Aldrin Piri
Hi Naveen, The treatment of that file as one FlowFile is as intended. If you wanted to break that out to separate FlowFiles, you could use the SplitText processor with one line per split and have a FlowFile per object. On Mon, Nov 9, 2015 at 4:27 PM, Madhire, Naveen < naveen.madh...@capitalone.c

Re: Collaboration?

2015-11-17 Thread Aldrin Piri
Corey has a lot of awesome points and best practices. >From a slightly different vantage point, many of my core and early experiences to NiFi were on teams of only two or three people where we handled some large volume flows for various groups and organizations. Typically, these flows took form fa

Re: MergeContent Processor

2015-11-19 Thread Aldrin Piri
Elli, Your understanding of the functionality is correct. There are a couple of criteria that drive when a bin is "done." In this case, if you establish the optional maximum properties, these drive in closing out sooner. That is if a max age is specified, and any of the bins have gone beyond that

Re: MergeContent Processor

2015-11-24 Thread Aldrin Piri
cessor to do is, even after min entries has been reached, wait a certain > period of time to see if there are more files. This is not max bin size - I > view that as a failsafe mechanism so that something doesn't stay in the > queue forever if the minimums are not reached. > &

Re: GetSFTP with dynamic file filter

2015-11-24 Thread Aldrin Piri
Michael, Are you using 0.3.0? If so, the functionality you are looking for is absent from that version. That requested functionality is what led to https://issues.apache.org/jira/browse/NIFI-673 and the introduction of List/FetchSFTP. I am not sure I have any good suggestions for 0.3.0 to achie

Re: ExecuteSQL Processor Usage

2015-12-05 Thread Aldrin Piri
Hi Indus, The data content is the body of the FlowFile that comes out of the success relationship. This data is structured in Avro format. If you haven't had the chance to give it a look over, the usage guide give some additional details concerning interaction with ExecuteSQL [1]. >From here, t

Re: The EndPoints of GetTwitter

2015-12-08 Thread Aldrin Piri
Bob, The Firehose stream is not publicly available via Twitter and requires special authorization to utilize [1]. As a result, this endpoint was not incorporated into the GetTwitter processor's properties and there is no way to provide the totality of the public feed in NiFi. [1] https://dev.twit

Re: The EndPoints of GetTwitter

2015-12-08 Thread Aldrin Piri
I know that had been the case, but thought that Twitter had ceased or was in the process of further curtailing that access. Additionally, it seems I fibbed about the Firehose, sorry for the misstep on that one. These endpoints are those exposed by the Hosebird Client that Twitter provides in thei

Re: The EndPoints of GetTwitter

2015-12-08 Thread Aldrin Piri
Main consideration is if the APIs aren't identical, this won't work out of the box. At this point, it is not so much a GetTwitter as it is Get that has their own schema. Anecdotally, I have a bit more than a foggy recollection that DataSift's API was accessed and possessed a different "schema" th

Re: Testing a nifi flow via junit

2016-01-04 Thread Aldrin Piri
Hello Vincent, This is something that does not exist and there have been a few threads on this topic [1][2]. Summarily, these tools do not currently exist due to the preference of using the interactive and real-time command and control over the flow as well as the increasing difficulty of maintai

Re: Upgrade 0.3.0 to 0.4.1

2016-01-04 Thread Aldrin Piri
This occurs as shortly after the release is official and the formula update gets incorporated into the main homebrew repository by their group of committers. 0.4.1 is available in the main repo, currently. Some older versions are also available in the homebrew-versions tap. These will likely be

Re: Upgrade 0.3.0 to 0.4.1

2016-01-04 Thread Aldrin Piri
, Christopher Hamm wrote: > brew update and brew doctor is my friend. > On Jan 4, 2016 10:11 PM, "Aldrin Piri" wrote: > >> This occurs as shortly after the release is official and the formula >> update gets incorporated into the main homebrew repository by their grou

  1   2   >