Re: Docker to deploy NiFi

Hi Stephane, I've been using Docker compose to form a NiFi cluster for testing purpose. It's not on dockerhub, but I've shared it as a Github project. [1] My knowledge of Docker is fairly limited, but hope this helps. 1) Although I haven't played with those yet, docker-compose has resource alloc

Re: API 1.0 questions

Hello, I was able to start/stop processors using 1.0 API. I am not sure if the API has changed from 0.x but, it seems you don't have to increment the revision. I used the same revision that is returned by NiFi. Here is an example (it's Javascript): https://github.com/ijokarumawak/nifi-deploy-proc

Re: how to loop workflows

Hello Alessio, I have an example NiFi template to loop workflow using counter attribute and NiFi expression: https://gist.github.com/ijokarumawak/01c4fd2d9291d3e74ec424a581659ca8 NiFi data-flow can be cyclic to loop the same flow-file until certain condition meets. Koji On Thu, Oct 20, 2016 at

Re: PutElasticsearchHttp error behaviour

Hello Gaspar, I looked at the PutElasticsearchHttp code, it distinguishes following two types of error: 1. Error occurs when the processor tries to connect Elasticsearch, but it couldn't. E.g. the specified Elasticsearch URL is not correct and java.net.ConnectException is thrown. When this happen

Re: DistributedMapCache

Hi Yari, Thanks for the great question. I looked at the DistributedMapCacheClient/Server code briefly, but there's no high availability support with NiFi cluster. As you figured it out, we need to point one of nodes IP address, in order to share the same Cache storage among nodes within the same c

Re: Failure to insert/update into SQL integer field

Hi Daniel, Sorry to hear that you've encountered the issue, and taking so long for you to get any feedback. PutSQL simply uses Integer.parseInt() method to convert attribute value to an Integer to map it to a integer table column. Integer.parseInt("0.0") threw a NumberFormatException, although we

Re: Failure to insert/update into SQL integer field

ribute value differently? > > Either that or the fact that I was writing to a local postgres db rather > than H2? > > -Daniel > > > On Nov 7, 2016 3:00 AM, "Koji Kawamura" wrote: >> >> Hi Daniel, >> >> Sorry to hear that you&#x

Re: NiFi versions and remote process groups

Hello Russell, You should be able to connect NiFi 1.0.0 to NiFi 0.7.1 via RemoteProcessGroup. Please check this Gist, I confirmed that it works: https://gist.github.com/ijokarumawak/37e428d5be8ce8031220f87c5ee9601c Did you wait for a while and refresh the canvas and RemoteProcessGroup on NiFI 1.0

Re: data flow from one s3 bucket to another

Hello Gop, Have you already found how to move data around S3 buckets? I hope you do. But just in case if you haven't yet, I wrote a simple NiFi flow and shared it in Gist: https://gist.github.com/ijokarumawak/26ff675039e252d177b1195f3576cf9a I misconfigured region and got an error once, but after

Re: Is it possible to filter and eliminate the values in the json data format using the nifi processors

Hi Manojkumar, To extract elements from a single JSON flow file, then EvaluateJsonPath or JoltTransformJSON would be able to do the job. Could you share the entire sample input JSON file (including the elements you'd like to filter out) ? Thanks, Koji On Sat, Nov 19, 2016 at 10:27 PM, Manojkumar

Re: Is it possible to filter and eliminate the values in the json data format using the nifi processors

:50 PM, Manojkumar Ravichandran wrote: > Hi koji, > > > I have tried the EvaluateJsonPath processor but can't able to produce the > required output,here the sample Input and required output json file has > been attached > > After that i want implement the same scenario

Re: Is it possible to filter and eliminate the values in the json data format using the nifi processors

uot;@.PersonalDetails": "[#2].PersonalDetails" } } } ] Please also look at this blog post for detail. http://ijokarumawak.github.io/nifi/2016/11/22/nifi-jolt/ Koji On Mon, Nov 21, 2016 at 10:34 PM, Koji Kawamura wrote: > Hi Manojkumar, > > Tha

Re: Clustering configuration error -- HTTPS hostname wrong

Hello Adam, This Exception indicates that the node's certificate didn't match with the requested hostname. It seems when a node replicated a request to another node in the cluster using the 'host1.foo.com', the target node responded with a server certificate which was generated for other hostname.

Re: How to integrate a custom protocol with Apache Nifi

Hi Kant, Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi has a set of processors for WebSocket since 1.1.0. It enables NiFi to act as a WebSocket client to communicate with a remote WebSocket server, or makes NiFi a WebSocket server so that remote clients access to it via WebSo

Re: How to integrate a custom protocol with Apache Nifi

7, 2016 at 1:23 AM, kant kodali wrote: >> >> Hi Koji, >> >> That is an awesome explanation! I expected processors for HTTP2 at very >> least since it is widely used ( the entire GRPC stack runs on that). I am >> not sure how easy or hard it is to build one? >&

Re: problem creating simple cluster

Hi Brian, Are those three node running on a same host using different port? Or running on different hosts? nifi.properties has nifi.cluster.node.address configuration, which is used by a NiFi node to tell how other NiFi nodes should access the node. If the property is blank, NiFi uses 'localhost'

Re: problem creating simple cluster

Sorry, I overlooked the nifi.properties settings you shared. Would you share what you can see on the NiFi "Cluster window", from right top Hamburger menu, especially the 'Node Address' column? Thanks, Koji On Thu, Dec 8, 2016 at 10:10 AM, Koji Kawamura wrote: > Hi Bri

Re: Capturing stdout *and* stderr of command executed by ExecuteStreamCommandProcessor

Hello A.B. I wrote a simple Java class which produces OOM error, and tested how ExecuteStreamCommand works. In the case that after the Java program writes some output, then encounters OOM, it sends an outgoing flow file to "output stream" containing data produced until the OOM. The OOM error mes

Re: ReplaceText and special characters

Hello, I did some experiments to see if I can append a carriage return. GenerateFlowFile generating 10 random bytes, followed by: 1. ReplaceText - Replacement Value: SHIFT-ENTER - Replacement Strategy: Append 2. ReplaceText - Replacement Value: ${literal(''):replaceFirst('','\r')} - Repla

Re: Remote Processor Group error: Unable to communicate with remote NiFi cluster

Hello Kevin, Sorry to hear that you got bitten by the issue, I think the 'Unable to find remote process group with id ...' is the same one I've encountered with NiFi 1.0.0. The issue has been solved since 1.1.0. https://issues.apache.org/jira/browse/NIFI-2687 So, if you're able to upgrade the clu

Re: Remote Processor Group error: Unable to communicate with remote NiFi cluster

> RemoteProcessGroup successfully, but intermittently it will stop. > > I will try the 1.1.0 update to test, or wait for the 1.1.1 update. This might > just solve my problem. > > Thanks for your advice, > > Kevin > > -Original Message- > From: Koji Kawamura [mailto:i

Re: Kafka offset different from Consume_Kafka_0_10 flowfile count

Hello Raf, If I understand it correctly, there are two topics, lets's say topic A and B, and the flow seems like: 1. ProduceKafka -> topic A 2. External process consumes from topic A, process data and push those to topic B 3. topic B -> ConsumeKafka I tested a flow looks like above, and the last

Re: How to get flowfile data payload into my Python Stream Callback?

Hello James, It seems that you need to pass an instance of a class extending InputStreamCallback. Here is an example that I derived from NiFi test case: https://gist.github.com/ijokarumawak/1df6d34cd1b2861eb6b7432ee7245ccd Hope this helps. Thanks! Koji On Mon, Jan 16, 2017 at 9:02 AM, James McM

Re: How backup works when flow.xml size more than max storage?

Hi Prabhu, Thanks for sharing your experience with flow file archiving. The case that a single flow.xml.gz file size exceeds archive.max.storage was not considered well when I implemented NIFI-2145. By looking at the code, it currently works as follows: 1. The original conf/flow.xml.gz (> 1MB) is

Re: How backup works when flow.xml size more than max storage?

it > should delete the old first file and save the latest file, so that the count > 10 is maintained. Here we can maintain the workflow properly and backup is > also achieved without confusing with max.time and max.storage. Only case is > that the disk size exceeds, we should notify

Re: How backup works when flow.xml size more than max storage?

and preserves the latest file till we restart the flow?? > If so what happens if the size is keep on increasing with respect to 500MB, > it will save archive based on count or only latest archive throughtout nifi > is in running status?? > > Many thanks > > On Thu, Jan 19, 201

Re: How backup works when flow.xml size more than max storage?

rote: > Hi Koji, > > Both simulation looks perfect. I was expected this exact behavior and it > matches my requirement, also it sounds logical. Shall I expect this changes > in next nifi release version?? > > > Thank you so much for this tremendous support. > > > On

Re: How to download HTTP Content with multiple years in GetHTTP Processor or any processor?

Hi Prabhu, GetHTTP doesn't take input relationship, so I'd recommend to use InvokeHTTP instead. With UpdateAttribute and RouteOnAttribute, you can create a loop in NiFi flow. An example is available here: https://gist.github.com/ijokarumawak/01c4fd2d9291d3e74ec424a581659ca8 The loop counter can

Re: nifi at AWS

Hi Mohammed, Which version of NiFi are you using? If it's 1.0.0 or later, you can choose 'HTTP' as 'Transport Protocol' in RemoteProcessGroup configuration in your local NiFi, this is what Andrew suggested earlier. With HTTP transport protocol, the local NiFi will use HTTP port (8080 in your case

Re: nifi at AWS

ad a local file to this > remote instance (using GetFile processor), then it fails to make the > transaction. > > Regards > > On Wed, Jan 25, 2017 at 9:51 PM, Koji Kawamura > wrote: >> >> Hi Mohammed, >> >> Which version of NiFi are you using

Re: Problem when using backpressure to distribute load over nodes in a cluster

Hi Bas, I think I was able to create a NiFi flow that works as you expected. Multiple GetSQS share workload using NiFi back pressure. Did you set Batch Size of GetSQS to 1? I've put a flow template and detailed description on this Gist. https://gist.github.com/ijokarumawak/4a9189ac630cf6cf6cd2d3

Re: Problem when using backpressure to distribute load over nodes in a cluster

Hi Bas, Sorry for the late reply. Thanks for the clarification, I over simplified the flow. As you experienced, NiFi back pressure is handled per relationship and as long as a relationship has room to receive new flow files, source processor is scheduled to run. I don't think there's an existing

Re: Problem when using backpressure to distribute load over nodes in a cluster

Hi Bas, It worked as expected (at least for me). In a processor, it's possible to transfer incoming FlowFile back to itself, so the processor can investigate the FlowFile and free to decide put it back or transfer it to other relationship. I've created a JIRA NIFI-3452, and submit a Pull Request

Re: ConsumeKafka processor erroring when held up by full queue

Hello Nick, First, I assume "had a queue back up" means have a queue being back-pressure. Sorry if that was different meaning. I was trying to reproduce by following flow: ConsumeKafka_0_10 -- success: Back Pressure Object Threshold = 10 -- UpdateAttribute (Stopped) Then I used ./bin/kafka

Re: http form contents

Hi Mohammed, I looked at HandleHttpRequest code, it turns out that it doesn't support 'multipart/form-data'. Supporting form encoded request is a legitimate improvement for NiFi. Other users has asked the same question before: http://apache-nifi-users-list.2361937.n4.nabble.com/Retrieve-files-from

Re: the zombie NiFi

Hi Chico, Although it looks longer to get started, but the log shows NiFi has started: > 2017-02-13 11:32:48,081 INFO [main] org.apache.nifi.web.server.JettyServer > NiFi has started. The UI is available at the following URLs: > 2017-02-13 11:32:48,082 INFO [main] org.apache.nifi.web.server.Jett

Re: Cluster instability due to java.net.SocketTimeoutException: Read timed out

Hello, The timeout happens when a node replicate a request to nodes to get a merged cluster wide response, so increasing nifi.cluster.node.read.timeout might help (in conf/nifi.properties). Request replication shouldn't timeout ideally, increasing nifi.web.jetty.threads might help to increase each

Re: ConsumeKafka processor erroring when held up by full queue

>> >>> > When back-pressure happens on the queue coming out of ConsumeKafka, >> >>> > this can last for longer than session.timeout.ms, and when the >> >>> > processors resumes executing it receives this error on the first >> >

Re: ConsumeKafka processor erroring when held up by full queue

> Hey Koji, thanks for putting in the time. > > I have not had a chance to start working on this myself and I certainly > support any effort to resolve it. I'll take a look at your branch and play > around with it. > > Thanks!, > Nick > > On Mon, Feb 20, 2017 at 2:30 A

Re: ConsumeKafka processor erroring when held up by full queue

eb 22, 2017 at 8:47 AM, Koji Kawamura wrote: > Hi Nick, > > I understand that. I will continue adding more code to iron out > implementation. > Please let me know if you find anything by looking at the code. I'd > like you to review the branch in detail once the PR is ready. &

Re: send http request to nifi input port

If the multipart contains only text data, it might be possible to use those processors Andy suggested. However, if we support multipart natively, it can be done much easier and safer. I created a JIRA for this improvement before, but haven't been able to find a time to implement it myself. If anyon

Re: send http request to nifi input port

avoid using "handlehttprequest" processor and allow users to > communicate with the remote NIFI instance both from WEB UI and/or NIFI > instance (e.g., site-to-site). > > Regards > > On Mon, Feb 27, 2017 at 6:54 PM, Koji Kawamura > wrote: >> >> If the

Re: failure handling question

Hi, Wondering if this might be helpful, but I just wanted to share. I've been working on adding a more generic error handling pattern for NiFi processors, that can handle exceptions by categorizing error types. As a part of work for the JIRA: Add "Rollback on Failure" property to PutHiveStreaming,

Re: configuration unit measurement inconsistencies

Hello Eric, Plurality doesn't matter, "s", "sec", "secs", "second", "seconds" work the same as implemented here: https://github.com/apache/nifi/blob/master/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/FormatUtils.java#L154 Thanks, Koji On Tue, May 9, 2017 at 1:57 PM, Erik Madrid w

Re: variable url for remote process group

Hi Mohammed, As RemoteProcessGroup and underlying Site-to-Site protocol maintains connectivity between client and server, it is not supported to change remote endpoint dynamically. HTTP processors may work but it's not cluster aware so you will need a Load balancer in front of those. If the numbe

Re: Convert output of ExtractMediaMetaData to JSON

Hello, Maybe you've already figure it out, but I think your AttributesToJSON processor's 'Destination' property is set to 'flowfile-attribute' (by default). If so, created JSON is written in 'JSONAttributes' attribute. If you'd like it to be as content of outgoing FlowFile, please use 'flowfile-co

Re: How to process files sequentially?

Hi Prabhu, I think you can use EnforceOrder processor which is available since 1.2.0, without Wait/Notify processor. Here is a sample flow I tested how it can be used for use-cases like yours: https://gist.github.com/ijokarumawak/7e6158460cfcb0b5911acefbb455edf0 Thanks, Koji On Fri, May 19, 201

Re: How to process files sequentially?

to give dynamic number of entries/bin for MergeContent due to currently > there is no expression language supported? > > On Fri, May 19, 2017 at 10:34 AM, Koji Kawamura > wrote: >> >> Hi Prabhu, >> >> I think you can use EnforceOrder processor which is available sinc

Re: How to process files sequentially?

g it in windows? > > On Fri, May 19, 2017 at 12:50 PM, Koji Kawamura > wrote: >> >> Hi Prabhu, >> >> I just used MergeContent to confirm test result. In your case, I >> thought the goal is sending queries to SQL Server in order so I think >> you don'

Re: some processors runs only once in NiFi

Hi Prabhu, Same as ListHDFS, GetHTTP uses ETAG HTTP header, and if server returns NOT_MODIFIED(304), it doesn't create output FlowFile. The screenshot indicates that GetHTTP runs 61 times but it only creates output FlowFile once because it's not modified. I believe that is what's happening. Than

Re: Add relationship to RouteOnAttribute via REST API?

Hello Adam, I think you need to execute two requests, the 1st is to add a dynamic property to RouteOnAttribute, and the 2nd is to create a connection from RouteOnAttribute. Here is an example for the 1st request adding a dynamic property using cURL command: curl -i -H 'Content-Type: application/

Re: Set priority to files based on date time value stored on attribute

Hi Manoj, I think EnforceOrder would not be useful in your case, as it expects the order to increases one by one (without skip). As Pierre suggested, I'd suggest using PriorityAttributePrioritizer. Thanks, Koji On Thu, Jun 8, 2017 at 3:50 PM, Pierre Villard wrote: > Hi Manoj, > > You may want o

Re: Set priority to files based on date time value stored on attribute

Hi Andre, Epoch format is indeed incremental, but EnforceOrder needs every number without skip. Unless the use-case guarantees incoming FlowFiles has a perfect set of incremental numbers in successful flow, EnforceOrder is not helpful. E.g. 1, 5, 4, 2, 3 can be enforced by EnforceOrder but 100, 50

Re: NiFi 1.1.1 & 1.2.0 with PostgreSql 9.5

Hello Raymond, Does the PutSQL has some number (probably 1) shown on its top right corner in NiFi UI? If so, the execution thread is stuck at some state, then looking at a thread dump may help to investigate where it stops. Could you share "$NIFI_HOME/bin/nifi.sh dump" result? The output will be w

Re: How to ensure the rows moved into SQL?

Hi Prabhu, In that case, Wait/Notify processors may be helpful. I've written a blog post before covering the similar use-case using those. http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/ Thanks, Koji On Tue, Jun 13, 2017 at 1:34 PM, prabhu Mahendran wrote: > Thanks for your res

Re: HDF NIfi - Does Nifi writes provenance/data on HDP Node ?

Hi Shashi, Sorry for delayed response. I am not aware that NiFi writes any provenance information on HDP nodes. But if your goal is to expose NiFi provenance data to HDFS, Hive (or Spark) to analyze provenance data using those services, then SiteToSiteProvenanceReportingTask might be helpful. Sit

Re: A newcomer Question about nifi use

Hi Hollis, To execute a remote shell using ssh, I'd use 'sh' command and its -c option with ExecuteStreamCommand or ExecuteProcess. For example, ExecuteStreamCommand configuration would look like: - Command Path: sh - Command Arguments: -c;ssh u...@host.example.com command-to-execute-on-remote -

Re: Safeguarding against List/Fetch of partial files

Hi James, Pierre, ListFile resets its state (including what is the latest entry it listed) when min file age is changed. ListFile.isListingResetNecessary implements the behavior. Thanks, Koji On Tue, Jul 18, 2017 at 2:42 AM, Pierre Villard wrote: > Hi James, > > This parameter should not change

Re: Safeguarding against List/Fetch of partial files

) || MAX_SIZE.equals(property) || IGNORE_HIDDEN_FILES.equals(property); On Tue, Jul 18, 2017 at 10:32 AM, Koji Kawamura wrote: > Hi James, Pierre, > > ListFile resets its state (including what is the latest entry it > listed) when min file age is changed. ListFile.isListingResetNecessary >

Re: Batch processing template

Hi Andy, I think using Wait processor to hold 5 files for a batch id is reasonable. A situation that may be problematic is that if more than one file for the same filetype are put into S3 mistakenly. To catch those illegal situations, Wait/Notify and RouteOnAttribute combination may be helpful. I

Re: Adding a Receive Date Time stamp to MQTT message.

Hi Bruce, By looking at the stacktrace, the exception complains that CSVRecordReader could not read record correctly at PutDatabaseRecord. Probably you need to use JsonTreeReader at PutDatabaseRecord since you used JsonRecordSetWriter at UpdateRecord. The error looks as if PutDatabaseRecord tried

Re: Getmongo processor query field

Hi Faustina, I'm not familiar with MongoDB query syntax, but based on the Stackoverflow answer, to avoid JsonParseException and query a field containing space, something like below may work? i.e. escaping the double quotes {"$where": "\"Incident Submitted Dt\" >= dd()"} Thanks, Koji On Wed, Se

Re: Getmongo processor query field

for your response. > The above code removed the JsonParseException, but it is fetching all the > data from the database. It's not executing my function call dd(). Is there a > way to make a function call in the same query? > > Thanks, > Faustina > > > On 6 September

Re: Getmongo processor query field

estamp into the > query. > > Mike > > On Wed, Sep 6, 2017 at 12:45 PM, faustina Selvadeepa > wrote: >> >> Thanks a ton koji, this worked :) >> >> Thanks, >> Faustina >> >> On 06-Sep-2017 2:20 PM, "Koji Kawamura" wrote: >> &

Re: Slow FTP and SFTP nifi transfer rates

Hello Gino, Thanks for sharing your findings on FTP performance. How did you measure send rate from NiFi to your FTP server? Sending multiple FlowFiles would provide less throughput compared to sending one big FlowFile, as PutFTP and PutSFTP make connection to each incoming FlowFile. The overhea

Re: Time-based Dependencies

Hi Andy, NiFi processors can be scheduled with CRON driven scheduling by specifying a time such as 5pm daily. Also, NiFi has different processors to control a FlowFile movement, such as ControlRate, Wait and Notify, EnforceOrder, or RouteOnAttribute ... etc. For example, UpdateAttribute and Route

Re: Slow FTP and SFTP nifi transfer rates

or 100 1GB files, the maximum throughput > rate of nifi PutFTP and PutSFTP remain the same. 300Mbps and 1Gbs > > As mention above, the weird thing is when I send files though ftp and sftp > (without nifi) then the rates are much better. > > It's really odd the the rates ar

Re: Slow FTP and SFTP nifi transfer rates

10MB file # BufferSize = 1KB (default) about 8 sec # BufferSize = 16KB about 300 ms I'm going to create a JIRA to add a processor property to specify buffer size. Also, will test SFTP. Thanks again for highlighting the issue! Koji On Fri, Sep 8, 2017 at 8:48 AM, Koji Kawamura wrote: >

Re: Slow FTP and SFTP nifi transfer rates

ice work koji and Gino! >> >> >> On Sep 8, 2017 6:54 AM, "Gino Lisignoli" wrote: >> >> Wow that sounds promising! would that also be the same for any other >> get/put processors? >> >> On Fri, Sep 8, 2017 at 7:47 PM

Re: QueryDatabaseTable - Schema

Hi Uwe, I had a similar expectation when I was using QueryDatabaseTable or any other processor creating Avro FlowFile which has its schema embedded, combining new record reader/writer controllers. Now, NiFi has "Inherit Record Schema" option as "Schema Access Strategy" of RecordWriter, already me

Re: Re: QueryDatabaseTable - Schema

enging things such as feeding kafka and maybe also > hadoop. So I am experimenting a lot and want to find the best possible > setup. > > Greetings ans thanks again. > > Uwe > > > Gesendet: Dienstag, 12. September 2017 um 03:05 Uhr > Von: "Koji Kawamura" &

Re: Removing duplicates from data

Hello Vikram, Welcome to NiFi and the community :) Would you elaborate your data flow? And which version you are using? For example, can you share some input data extracted from Oracle? I wonder why you need to remove duplicate records while PostgreSQL doesn't have primary key constraint, or why

Re: Removing duplicates from data

ble has been model/designed by >> us (and can add primary key). Data from oracle looks like below (i need to >> remove duplicate record for combination on ColA , ColB) >> >> Col A Col B >> C1 item 1 >> C1 item 2 >> *C2* *item 3* >> *C2* *item 4* &

Re: [EXT] New to Nifi - Failed to update database due to a failed batch update

Hi Aruna, The placeholders in your ReplaceText configuration, such as '${city_name}' are NiFi Expression Language. If the incoming FlowFile has such FlowFile Attributes, those can be replaced with FlowFile Attribute values. But I suspect FlowFile doesn't have those attributes since ReplaceText is

Re: [EXT] New to Nifi - Failed to update database due to a failed batch update

hese examples useful. Thanks, Koji On Tue, Sep 26, 2017 at 11:23 PM, Aruna Sankaralingam < aruna.sankaralin...@cormac-corp.com> wrote: > I am not sure I understand. This is how my CSV looks. > > > > > > -Original Message- > From: Koji Kawamura [mailto:ij

Re: [EXT] New to Nifi - Failed to update database due to a failed batch update

s > well. How do I find out what is wrong? > > > > > > *From:* Koji Kawamura [mailto:ijokaruma...@gmail.com] > *Sent:* Tuesday, September 26, 2017 10:22 PM > > *To:* users@nifi.apache.org > *Cc:* karthi keyan > *Subject:* Re: [EXT] New to Nifi - Failed to update databas

Re: Use Nifi Secure S2S with proxy

Hi Ali, A single forward proxy server can be a SPOF. Although I haven't tried myself, you should be able to make it highly available by deploying multiple ones and a LB in front of those (such as Squid proxies behind HA proxy, I found couple of blog posts about this configuration). As long as each

Re: Use Nifi Secure S2S with proxy

forward proxy? > > On Tue, Oct 10, 2017 at 2:29 PM, Koji Kawamura > wrote: >> >> Hi Ali, >> >> A single forward proxy server can be a SPOF. Although I haven't tried >> myself, you should be able to make it highly available by deploying >> multiple

Re: Transformations using Nifi

Hi Aruna, If you can not upgrade from NiFi 1.2.0, then I think the best bed is using: ScriptedReader or ScriptedRecordSetWriter for data conversion #1, 2 and 3. As Russ mentioned, EL might be helpful when you implement the scripted components. #4 is a bit harder since it requires a database conne

Re: EvaluateJsonPath Processor Resulting EmptyStringSet For Flow Attribute

Hi Rakesh, Is the screenshot URL correct? I can only see controller services table view. http://apache-nifi-users-list.2361937.n4.nabble.com/file/t310/NifiHelp.png Would you share EvaluateJsonPath processor configuration? Thanks, Koji On Mon, Oct 16, 2017 at 3:39 PM, rakesh wrote: > Hi Team, >

Re: SplitAvro fails to split data

Hi Pradip, When I tested the scenario with NiFi 1.4, SplitAvro created an outgoing FlowFile containing single record as expected when the incoming FlowFile only has one Avro record. In SplitAvro change histories, I found NIFI-2841. I assume the change fixed the split behavior with single input re

Re: NiFi ExecuteSQL error => can not be represented as java.sql.Timestamp

Hi Mohit, The exception looks as if the entire string ' 821725069 2161514622096 ... 0-00 0 3' was converted to java.sql.Timestamp. Would you share your create table DDL statement, few sample record data, NiFi, MySQL and JDBC driver version you're using? Thanks, Koji On Wed

Re: NiFi ExecuteSQL error => can not be represented as java.sql.Timestamp

> jdbc:mysql://localhost:3306/nifi_test?zeroDateTimeBehavior=convertToNull > > It was basically causing due to null values in a Timestamp column which I was > able to cater by converting it to null. > > Thanks, > Mohit > > -Original Message- > From: Koji Kawamura

Re: [EXT] Re: Wait only if flagged?

Hi Peter, I think I have done similar thing before. This flow template may be useful in your case, too. https://gist.github.com/ijokarumawak/9e1a4855934f2bb9661f88ca625bd244 Thanks, Koji On Tue, Nov 14, 2017 at 12:58 PM, Peter Wicks (pwicks) wrote: > Matt, > > I played around with your idea. I

Re: Read incremental records from DB

Hi Siva, There was a similar question before, and Matt answered with an example flow template. http://apache-nifi-developer-list.39713.n7.nabble.com/Example-of-GenerateTableFetch-td14374.html You can use ExecuteSQL to execute the generated SQLs by GenerateTableFetch. Since ExecuteSQL returns sele

Re: GetSFTP error

Hello, I haven't tried it myself, but from the stacktrace and Jsch souce code, I think you should specify a file in pkcs8 format, instead of pkcs12. Jsch will leave keypair null if it fails to parse it, that may be the cause of the NullPointerException. For converting a pem file to a pkcs8, there

Re: Clarification on load distribution on NiFi cluster

Hi Ravi, To distribute QueryDatabaseTable workload, I'd suggest using GenerateTableFetch instead. Because it can generate SQLs to query updated records. And those SQL FlowFiles can be distributed among NiFi nodes by RPG. - Following lines are just to share my thoughts on the topic for possib

Re: Replay Event UUID

Hi Rotem, When I tested it with NiFi 1.5.0-SNAPSHOT, the REPLAY event has its FlowFile UUID as the parent (original) FlowFile UUID as expected. Type REPLAY FlowFile Uuid 8c61fdd7-c084-4756-946c-f5669dc4442d File Size 4 bytes Component Id 9abc21e3-0160-1000-6d6f-a1c408f75b7a Component Name UpdateA

Re: Bug/Unexpected behavior in ConvertJSONToSQL for boolean attributes

Hi Jennifer, Thank you very much for reporting this. It seems the line converts a Boolean to "0" or "1" at ConvertJSONToSQL is implemented wrongly. Looks like a careless mistake. Sorry for the inconvenience. https://github.com/apache/nifi/commit/8acee02393f9557b9679038b933ba49705984cf8#diff-d99a1a

Re: Bug/Unexpected behavior in ConvertJSONToSQL for boolean attributes

uteSQL and ConvertRecord 1 -> true, 0 -> false) using following NiFi flow. https://gist.github.com/ijokarumawak/5b8d7dd5d799764dfd13dc6195025785 I hope this to get merged soon and available in the next release. Thanks, Koji On Wed, Jan 3, 2018 at 8:13 AM, Koji Kawamura wrote: > Hi Jenn

Re: Replay Event UUID

anged to the WriteAheadProvenanceRepository (like on the server where we > first noticed this) we did see it. Have there been changes in that > implementation? Could this cause what we are seeing? > > Thanks! > Rotem > > > On 28 Dec 2017 3:35 am, "Koji Kawamura" w

Re: ListS3 and FetchS3

Hi Aruna, The resulted two FlowFiles have the same contents I guess, that is the PDF file you specified at FetchS3Object Object Key. The flow worked as follows actually: 1. ListS3 listed two FlowFiles, Ntl_15.csv and 11500509.pdf 2. FetchS3Object is executed twice for each incoming FlowFile 2-1. F

Re: Get the failure reason from ValidateResult processor

Hi Martin, I assume you wanted to ask about ValidateRecord. As you know, ValidateRecord processor emits ROUTE provenance events with 'Details' that explains validation error. E.g. "Records in this FlowFile were invalid for the following reasons: ; The following 1 fields were present in the Record

Re: Maximum-value Columns on QueryDatabaseTable

Hi Alberto, Thanks for reporting the issue, I was able to reproduce the behavior you described. Although it's for Microsoft SQL Server, there has been an existing JIRA for the same issue, NIFI-4393. https://issues.apache.org/jira/browse/NIFI-4393 I've created a Pull Request to fix MS SQL square b

Re: Updating schedule information using REST API without disturbing other processor configuration

Hi Ravi, How does your request JSON sent to the PUT /processors/{id} endpoint look like? If you don't need to update any processor properties, then you don't have to send /component/config/properties element in a request JSON. You can debug how NiFi UI sends REST requests using web browser Develop

Re: all of our schedule tasks not running/being scheduled....

Hi Dan, If all available Timer Driven Thread are being used (or hang unexpectedly for some reason), then no processor can be scheduled. The number at the left top the NiFi UI under the NiFi logo shows the number of threads currently working. If you see something more than 0, then I'd recommend to

Re: adding a filename column to a csv to insert into a table

Hi Austin, I think there are a couple of ways to do that: 1. UpdateRecord with CSVReader and CSVWriter, update a column with a Record Path and Expression Language, e.g. Add a dynamic property, key=/filename, value=${filename} 2. Use SpritText to sprit each CSV record into a FlowFile, then combine

Re: Is it possible to join multiple columns to a record using single lookup service

Hi Sangavi, Good question, I thought it can be a nice example to illustrate how to use LookupService. I wrote a simple Gist entry with a NiFi template file to do what you are looking for. "Join, Enrich multiple columns by looking up an external CSV file" https://gist.github.com/ijokarumawak/b9c95

Re: Secure NiFi 1.5 Behind NGINX/HAProxy

Hi Ryan, Although I am not sure why you'd want to use http between the clients and Nginx, I was able to setup similar environment. I used LDAP provider instead of OpenID, but OpenID should work as well. The key is NOT provide any client certificate from clients (browser/API) and Nginx to NiFi, so

Re: Create nested records

Hi Charlie, Thanks for sharing the template. Following configurations for UpdateRecord did the flat to nested mapping: - Replacement Value Strategy: Record Path Value - Dynamic property: "/phone" = "/" It maps the flat record into /phone child record. Fields those are not included in the outgoin

  1   2   >