Re: Read incremental records from DB

Hi Siva, There was a similar question before, and Matt answered with an example flow template. http://apache-nifi-developer-list.39713.n7.nabble.com/Example-of-GenerateTableFetch-td14374.html You can use ExecuteSQL to execute the generated SQLs by GenerateTableFetch. Since ExecuteSQL returns

Re: [EXT] Re: Wait only if flagged?

Hi Peter, I think I have done similar thing before. This flow template may be useful in your case, too. https://gist.github.com/ijokarumawak/9e1a4855934f2bb9661f88ca625bd244 Thanks, Koji On Tue, Nov 14, 2017 at 12:58 PM, Peter Wicks (pwicks) wrote: > Matt, > > I played

Re: NiFi ExecuteSQL error => can not be represented as java.sql.Timestamp

ng with my JDBC connection - > > jdbc:mysql://localhost:3306/nifi_test?zeroDateTimeBehavior=convertToNull > > It was basically causing due to null values in a Timestamp column which I was > able to cater by converting it to null. > > Thanks, > Mohit > > ---

Re: NiFi ExecuteSQL error => can not be represented as java.sql.Timestamp

Hi Mohit, The exception looks as if the entire string ' 821725069 2161514622096 ... 0-00 0 3' was converted to java.sql.Timestamp. Would you share your create table DDL statement, few sample record data, NiFi, MySQL and JDBC driver version you're using? Thanks, Koji On

Re: SplitAvro fails to split data

Hi Pradip, When I tested the scenario with NiFi 1.4, SplitAvro created an outgoing FlowFile containing single record as expected when the incoming FlowFile only has one Avro record. In SplitAvro change histories, I found NIFI-2841. I assume the change fixed the split behavior with single input

Re: EvaluateJsonPath Processor Resulting EmptyStringSet For Flow Attribute

Hi Rakesh, Is the screenshot URL correct? I can only see controller services table view. http://apache-nifi-users-list.2361937.n4.nabble.com/file/t310/NifiHelp.png Would you share EvaluateJsonPath processor configuration? Thanks, Koji On Mon, Oct 16, 2017 at 3:39 PM, rakesh

Re: Use Nifi Secure S2S with proxy

ing with any forward proxy? > > On Tue, Oct 10, 2017 at 2:29 PM, Koji Kawamura <ijokaruma...@gmail.com> > wrote: >> >> Hi Ali, >> >> A single forward proxy server can be a SPOF. Although I haven't tried >> myself, you should be able to make it highly available

Re: Use Nifi Secure S2S with proxy

Hi Ali, A single forward proxy server can be a SPOF. Although I haven't tried myself, you should be able to make it highly available by deploying multiple ones and a LB in front of those (such as Squid proxies behind HA proxy, I found couple of blog posts about this configuration). As long as

Re: [EXT] New to Nifi - Failed to update database due to a failed batch update

I find out what is wrong? > > > > > > *From:* Koji Kawamura [mailto:ijokaruma...@gmail.com] > *Sent:* Tuesday, September 26, 2017 10:22 PM > > *To:* users@nifi.apache.org > *Cc:* karthi keyan > *Subject:* Re: [EXT] New to Nifi - Failed to update database due to a > fail

Re: [EXT] New to Nifi - Failed to update database due to a failed batch update

. Thanks, Koji On Tue, Sep 26, 2017 at 11:23 PM, Aruna Sankaralingam < aruna.sankaralin...@cormac-corp.com> wrote: > I am not sure I understand. This is how my CSV looks. > > > > > > -Original Message- > From: Koji Kawamura [mailto:ijokaruma...@gmail.com

Re: [EXT] New to Nifi - Failed to update database due to a failed batch update

Hi Aruna, The placeholders in your ReplaceText configuration, such as '${city_name}' are NiFi Expression Language. If the incoming FlowFile has such FlowFile Attributes, those can be replaced with FlowFile Attribute values. But I suspect FlowFile doesn't have those attributes since ReplaceText is

Re: Removing duplicates from data

gt;> C3 item 1 >> C4 null >> C5 item 5 >> C6 item 7 >> I will try to explore PutDatabaseRecord processor and see i can achieve >> desired purpose. >> >> Thanks, >> Vikram >> >> On Mon, Sep 18, 2017 at 9:59 PM, Koji Kawamura <ijokaruma...@

Re: Re: QueryDatabaseTable - Schema

ng kafka and maybe also > hadoop. So I am experimenting a lot and want to find the best possible > setup. > > Greetings ans thanks again. > > Uwe > > > Gesendet: Dienstag, 12. September 2017 um 03:05 Uhr > Von: "Koji Kawamura" <ijokaruma...@gmail.com&g

Re: QueryDatabaseTable - Schema

Hi Uwe, I had a similar expectation when I was using QueryDatabaseTable or any other processor creating Avro FlowFile which has its schema embedded, combining new record reader/writer controllers. Now, NiFi has "Inherit Record Schema" option as "Schema Access Strategy" of RecordWriter, already

Re: Slow FTP and SFTP nifi transfer rates

# BufferSize = 1KB (default) about 8 sec # BufferSize = 16KB about 300 ms I'm going to create a JIRA to add a processor property to specify buffer size. Also, will test SFTP. Thanks again for highlighting the issue! Koji On Fri, Sep 8, 2017 at 8:48 AM, Koji Kawamura <ijokaruma...@gmail.

Re: Slow FTP and SFTP nifi transfer rates

maximum throughput > rate of nifi PutFTP and PutSFTP remain the same. 300Mbps and 1Gbs > > As mention above, the weird thing is when I send files though ftp and sftp > (without nifi) then the rates are much better. > > It's really odd the the rates are significantly slower in NIFI. >

Re: Time-based Dependencies

Hi Andy, NiFi processors can be scheduled with CRON driven scheduling by specifying a time such as 5pm daily. Also, NiFi has different processors to control a FlowFile movement, such as ControlRate, Wait and Notify, EnforceOrder, or RouteOnAttribute ... etc. For example, UpdateAttribute and

Re: Slow FTP and SFTP nifi transfer rates

Hello Gino, Thanks for sharing your findings on FTP performance. How did you measure send rate from NiFi to your FTP server? Sending multiple FlowFiles would provide less throughput compared to sending one big FlowFile, as PutFTP and PutSFTP make connection to each incoming FlowFile. The

Re: Getmongo processor query field

ng EL to inject a raw timestamp into the > query. > > Mike > > On Wed, Sep 6, 2017 at 12:45 PM, faustina Selvadeepa <faustin...@gmail.com> > wrote: >> >> Thanks a ton koji, this worked :) >> >> Thanks, >> Faustina >> >> On

Re: Getmongo processor query field

Hi Faustina, I'm not familiar with MongoDB query syntax, but based on the Stackoverflow answer, to avoid JsonParseException and query a field containing space, something like below may work? i.e. escaping the double quotes {"$where": "\"Incident Submitted Dt\" >= dd()"} Thanks, Koji On Wed,

Re: Adding a Receive Date Time stamp to MQTT message.

Hi Bruce, By looking at the stacktrace, the exception complains that CSVRecordReader could not read record correctly at PutDatabaseRecord. Probably you need to use JsonTreeReader at PutDatabaseRecord since you used JsonRecordSetWriter at UpdateRecord. The error looks as if PutDatabaseRecord tried

Re: Batch processing template

Hi Andy, I think using Wait processor to hold 5 files for a batch id is reasonable. A situation that may be problematic is that if more than one file for the same filetype are put into S3 mistakenly. To catch those illegal situations, Wait/Notify and RouteOnAttribute combination may be helpful.

Re: Safeguarding against List/Fetch of partial files

) || MAX_SIZE.equals(property) || IGNORE_HIDDEN_FILES.equals(property); On Tue, Jul 18, 2017 at 10:32 AM, Koji Kawamura <ijokaruma...@gmail.com> wrote: > Hi James, Pierre, > > ListFile resets its state (including what is the latest entry it > listed) when min file age is changed. ListFile.isLis

Re: A newcomer Question about nifi use

Hi Hollis, To execute a remote shell using ssh, I'd use 'sh' command and its -c option with ExecuteStreamCommand or ExecuteProcess. For example, ExecuteStreamCommand configuration would look like: - Command Path: sh - Command Arguments: -c;ssh u...@host.example.com command-to-execute-on-remote -

Re: HDF NIfi - Does Nifi writes provenance/data on HDP Node ?

Hi Shashi, Sorry for delayed response. I am not aware that NiFi writes any provenance information on HDP nodes. But if your goal is to expose NiFi provenance data to HDFS, Hive (or Spark) to analyze provenance data using those services, then SiteToSiteProvenanceReportingTask might be helpful.

Re: How to ensure the rows moved into SQL?

Hi Prabhu, In that case, Wait/Notify processors may be helpful. I've written a blog post before covering the similar use-case using those. http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/ Thanks, Koji On Tue, Jun 13, 2017 at 1:34 PM, prabhu Mahendran

Re: NiFi 1.1.1 & 1.2.0 with PostgreSql 9.5

Hello Raymond, Does the PutSQL has some number (probably 1) shown on its top right corner in NiFi UI? If so, the execution thread is stuck at some state, then looking at a thread dump may help to investigate where it stops. Could you share "$NIFI_HOME/bin/nifi.sh dump" result? The output will be

Re: Set priority to files based on date time value stored on attribute

Hi Andre, Epoch format is indeed incremental, but EnforceOrder needs every number without skip. Unless the use-case guarantees incoming FlowFiles has a perfect set of incremental numbers in successful flow, EnforceOrder is not helpful. E.g. 1, 5, 4, 2, 3 can be enforced by EnforceOrder but 100,

Re: Set priority to files based on date time value stored on attribute

Hi Manoj, I think EnforceOrder would not be useful in your case, as it expects the order to increases one by one (without skip). As Pierre suggested, I'd suggest using PriorityAttributePrioritizer. Thanks, Koji On Thu, Jun 8, 2017 at 3:50 PM, Pierre Villard wrote:

Re: Add relationship to RouteOnAttribute via REST API?

Hello Adam, I think you need to execute two requests, the 1st is to add a dynamic property to RouteOnAttribute, and the 2nd is to create a connection from RouteOnAttribute. Here is an example for the 1st request adding a dynamic property using cURL command: curl -i -H 'Content-Type:

Re: How to process files sequentially?

; > How to give dynamic number of entries/bin for MergeContent due to currently > there is no expression language supported? > > On Fri, May 19, 2017 at 10:34 AM, Koji Kawamura <ijokaruma...@gmail.com> > wrote: >> >> Hi Prabhu, >> >> I think you can u

Re: How to process files sequentially?

Hi Prabhu, I think you can use EnforceOrder processor which is available since 1.2.0, without Wait/Notify processor. Here is a sample flow I tested how it can be used for use-cases like yours: https://gist.github.com/ijokarumawak/7e6158460cfcb0b5911acefbb455edf0 Thanks, Koji On Fri, May 19,

Re: Convert output of ExtractMediaMetaData to JSON

Hello, Maybe you've already figure it out, but I think your AttributesToJSON processor's 'Destination' property is set to 'flowfile-attribute' (by default). If so, created JSON is written in 'JSONAttributes' attribute. If you'd like it to be as content of outgoing FlowFile, please use

Re: variable url for remote process group

Hi Mohammed, As RemoteProcessGroup and underlying Site-to-Site protocol maintains connectivity between client and server, it is not supported to change remote endpoint dynamically. HTTP processors may work but it's not cluster aware so you will need a Load balancer in front of those. If the

Re: configuration unit measurement inconsistencies

Hello Eric, Plurality doesn't matter, "s", "sec", "secs", "second", "seconds" work the same as implemented here: https://github.com/apache/nifi/blob/master/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/FormatUtils.java#L154 Thanks, Koji On Tue, May 9, 2017 at 1:57 PM, Erik Madrid

Re: failure handling question

Hi, Wondering if this might be helpful, but I just wanted to share. I've been working on adding a more generic error handling pattern for NiFi processors, that can handle exceptions by categorizing error types. As a part of work for the JIRA: Add "Rollback on Failure" property to

Re: send http request to nifi input port

I think, > this will avoid using "handlehttprequest" processor and allow users to > communicate with the remote NIFI instance both from WEB UI and/or NIFI > instance (e.g., site-to-site). > > Regards > > On Mon, Feb 27, 2017 at 6:54 PM, Koji Kawamura <ijokaruma...@gmail.com

Re: send http request to nifi input port

If the multipart contains only text data, it might be possible to use those processors Andy suggested. However, if we support multipart natively, it can be done much easier and safer. I created a JIRA for this improvement before, but haven't been able to find a time to implement it myself. If

Re: ConsumeKafka processor erroring when held up by full queue

, 2017 at 8:47 AM, Koji Kawamura <ijokaruma...@gmail.com> wrote: > Hi Nick, > > I understand that. I will continue adding more code to iron out > implementation. > Please let me know if you find anything by looking at the code. I'd > like you to review the branch in det

Re: ConsumeKafka processor erroring when held up by full queue

> On Mon, Feb 20, 2017 at 2:30 AM, Koji Kawamura <ijokaruma...@gmail.com> > wrote: >> >> Hi Nick, Joe, Bryan, >> >> I confirmed that this is easily reproducible and I got exactly the >> same stacktrace. >> >> Also I was curious about how Kafka consume

Re: ConsumeKafka processor erroring when held up by full queue

ger than session.timeout.ms, and when the >> >>> > processors resumes executing it receives this error on the first >> >>> > execution. We should be able to implement some type of keep-alive so >> >>> > that even when the processor is not executin

Re: Cluster instability due to java.net.SocketTimeoutException: Read timed out

Hello, The timeout happens when a node replicate a request to nodes to get a merged cluster wide response, so increasing nifi.cluster.node.read.timeout might help (in conf/nifi.properties). Request replication shouldn't timeout ideally, increasing nifi.web.jetty.threads might help to increase

Re: the zombie NiFi

Hi Chico, Although it looks longer to get started, but the log shows NiFi has started: > 2017-02-13 11:32:48,081 INFO [main] org.apache.nifi.web.server.JettyServer > NiFi has started. The UI is available at the following URLs: > 2017-02-13 11:32:48,082 INFO [main]

Re: http form contents

Hi Mohammed, I looked at HandleHttpRequest code, it turns out that it doesn't support 'multipart/form-data'. Supporting form encoded request is a legitimate improvement for NiFi. Other users has asked the same question before:

Re: ConsumeKafka processor erroring when held up by full queue

Hello Nick, First, I assume "had a queue back up" means have a queue being back-pressure. Sorry if that was different meaning. I was trying to reproduce by following flow: ConsumeKafka_0_10 -- success: Back Pressure Object Threshold = 10 -- UpdateAttribute (Stopped) Then I used

Re: Problem when using backpressure to distribute load over nodes in a cluster

Hi Bas, It worked as expected (at least for me). In a processor, it's possible to transfer incoming FlowFile back to itself, so the processor can investigate the FlowFile and free to decide put it back or transfer it to other relationship. I've created a JIRA NIFI-3452, and submit a Pull

Re: Problem when using backpressure to distribute load over nodes in a cluster

Hi Bas, Sorry for the late reply. Thanks for the clarification, I over simplified the flow. As you experienced, NiFi back pressure is handled per relationship and as long as a relationship has room to receive new flow files, source processor is scheduled to run. I don't think there's an

Re: nifi at AWS

Hi Mohammed, Which version of NiFi are you using? If it's 1.0.0 or later, you can choose 'HTTP' as 'Transport Protocol' in RemoteProcessGroup configuration in your local NiFi, this is what Andrew suggested earlier. With HTTP transport protocol, the local NiFi will use HTTP port (8080 in your

Re: How to download HTTP Content with multiple years in GetHTTP Processor or any processor?

Hi Prabhu, GetHTTP doesn't take input relationship, so I'd recommend to use InvokeHTTP instead. With UpdateAttribute and RouteOnAttribute, you can create a loop in NiFi flow. An example is available here: https://gist.github.com/ijokarumawak/01c4fd2d9291d3e74ec424a581659ca8 The loop counter can

Re: How backup works when flow.xml size more than max storage?

rt. > > > On Fri, Jan 20, 2017 at 6:14 AM, Koji Kawamura <ijokaruma...@gmail.com> > wrote: >> >> Hi Prabhu, >> >> In that case, yes, as your assumption, even the latest archive exceeds >> 500MB, the latest archive is saved, as long as it was written to d

Re: How backup works when flow.xml size more than max storage?

rease the size and preserves the latest file till we restart the flow?? > If so what happens if the size is keep on increasing with respect to 500MB, > it will save archive based on count or only latest archive throughtout nifi > is in running status?? > > Many thanks > >

Re: How backup works when flow.xml size more than max storage?

, we should notify user about this. > > > Many thanks. > > > On Thu, Jan 19, 2017 at 6:36 AM, Koji Kawamura <ijokaruma...@gmail.com> > wrote: >> >> Hi Prabhu, >> >> Thanks for sharing your experience with flow file archiving. >> The case that a s

Re: How backup works when flow.xml size more than max storage?

Hi Prabhu, Thanks for sharing your experience with flow file archiving. The case that a single flow.xml.gz file size exceeds archive.max.storage was not considered well when I implemented NIFI-2145. By looking at the code, it currently works as follows: 1. The original conf/flow.xml.gz (> 1MB)

Re: How to get flowfile data payload into my Python Stream Callback?

Hello James, It seems that you need to pass an instance of a class extending InputStreamCallback. Here is an example that I derived from NiFi test case: https://gist.github.com/ijokarumawak/1df6d34cd1b2861eb6b7432ee7245ccd Hope this helps. Thanks! Koji On Mon, Jan 16, 2017 at 9:02 AM, James

Re: Remote Processor Group error: Unable to communicate with remote NiFi cluster

es have moved through the > RemoteProcessGroup successfully, but intermittently it will stop. > > I will try the 1.1.0 update to test, or wait for the 1.1.1 update. This might > just solve my problem. > > Thanks for your advice, > > Kevin > > -Original Message- &

Re: problem creating simple cluster

Sorry, I overlooked the nifi.properties settings you shared. Would you share what you can see on the NiFi "Cluster window", from right top Hamburger menu, especially the 'Node Address' column? Thanks, Koji On Thu, Dec 8, 2016 at 10:10 AM, Koji Kawamura <ijokaruma...@gmail.com> w

Re: problem creating simple cluster

Hi Brian, Are those three node running on a same host using different port? Or running on different hosts? nifi.properties has nifi.cluster.node.address configuration, which is used by a NiFi node to tell how other NiFi nodes should access the node. If the property is blank, NiFi uses

Re: How to integrate a custom protocol with Apache Nifi

ed, Dec 7, 2016 at 1:23 AM, kant kodali <kanth...@gmail.com> wrote: >> >> Hi Koji, >> >> That is an awesome explanation! I expected processors for HTTP2 at very >> least since it is widely used ( the entire GRPC stack runs on that). I am >> not sure how

Re: Clustering configuration error -- HTTPS hostname wrong

Hello Adam, This Exception indicates that the node's certificate didn't match with the requested hostname. It seems when a node replicated a request to another node in the cluster using the 'host1.foo.com', the target node responded with a server certificate which was generated for other

Re: Is it possible to filter and eliminate the values in the json data format using the nifi processors

.PersonalDetails": "[#2].PersonalDetails" } } } ] Please also look at this blog post for detail. http://ijokarumawak.github.io/nifi/2016/11/22/nifi-jolt/ Koji On Mon, Nov 21, 2016 at 10:34 PM, Koji Kawamura <ijokaruma...@gmail.com> wrote: > Hi Manojkuma

Re: Is it possible to filter and eliminate the values in the json data format using the nifi processors

Hi Manojkumar, To extract elements from a single JSON flow file, then EvaluateJsonPath or JoltTransformJSON would be able to do the job. Could you share the entire sample input JSON file (including the elements you'd like to filter out) ? Thanks, Koji On Sat, Nov 19, 2016 at 10:27 PM,

Re: data flow from one s3 bucket to another

Hello Gop, Have you already found how to move data around S3 buckets? I hope you do. But just in case if you haven't yet, I wrote a simple NiFi flow and shared it in Gist: https://gist.github.com/ijokarumawak/26ff675039e252d177b1195f3576cf9a I misconfigured region and got an error once, but

Re: NiFi versions and remote process groups

Hello Russell, You should be able to connect NiFi 1.0.0 to NiFi 0.7.1 via RemoteProcessGroup. Please check this Gist, I confirmed that it works: https://gist.github.com/ijokarumawak/37e428d5be8ce8031220f87c5ee9601c Did you wait for a while and refresh the canvas and RemoteProcessGroup on NiFI

Re: Failure to insert/update into SQL integer field

he attribute value differently? > > Either that or the fact that I was writing to a local postgres db rather > than H2? > > -Daniel > > > On Nov 7, 2016 3:00 AM, "Koji Kawamura" <ijokaruma...@gmail.com> wrote: >> >> Hi Daniel, >> >> So

Re: Failure to insert/update into SQL integer field

Hi Daniel, Sorry to hear that you've encountered the issue, and taking so long for you to get any feedback. PutSQL simply uses Integer.parseInt() method to convert attribute value to an Integer to map it to a integer table column. Integer.parseInt("0.0") threw a NumberFormatException, although

Re: DistributedMapCache

Hi Yari, Thanks for the great question. I looked at the DistributedMapCacheClient/Server code briefly, but there's no high availability support with NiFi cluster. As you figured it out, we need to point one of nodes IP address, in order to share the same Cache storage among nodes within the same

Re: PutElasticsearchHttp error behaviour

Hello Gaspar, I looked at the PutElasticsearchHttp code, it distinguishes following two types of error: 1. Error occurs when the processor tries to connect Elasticsearch, but it couldn't. E.g. the specified Elasticsearch URL is not correct and java.net.ConnectException is thrown. When this

Re: how to loop workflows

Hello Alessio, I have an example NiFi template to loop workflow using counter attribute and NiFi expression: https://gist.github.com/ijokarumawak/01c4fd2d9291d3e74ec424a581659ca8 NiFi data-flow can be cyclic to loop the same flow-file until certain condition meets. Koji On Thu, Oct 20, 2016 at

Re: API 1.0 questions

Hello, I was able to start/stop processors using 1.0 API. I am not sure if the API has changed from 0.x but, it seems you don't have to increment the revision. I used the same revision that is returned by NiFi. Here is an example (it's Javascript):

Re: Upgrade 0.7.0 to 1.0.0

Hello Kumiko, Sorry to hear that you're having issues with upgrading to 1.0.0. 1) Any NiFi API URL of a node in the remote cluster should work for the target URL. If there're node1, 2, 3 in a remote cluster then "http://node<1, 2 or 3>:/nifi" should work. Every node is capable to handle web

Re: Approaches to Array in Json with Nifi?

Hi all, I got interested in EvaluateJsonPath, too. So I tried to figure out how to do that, and found there're few obstacles now. Input Json = {"company":"xyz", "rate":[0.02, 0.03, 0.04]} Then, applying following JsonPaths return: $.rate.min() => 0.2 $.rate.length() => 3 $.rate.avg() => 0.03

<    1   2