Re: Clustered Site-toSite

I am not following why you set all your Nodes (source and destination) to use the same hostname(s). Each hostname resolves to a single IP and by doing so doesn't all data get sent to a single end-point? The idea behind spreading out the connections when using S2S is for smart load balancing

[ANNOUNCE] CFP open for ApacheCon North America 2016

Community growth starts by talking with those interested in your project. ApacheCon North America is coming, are you? We are delighted to announce that the Call For Presentations (CFP) is now open for ApacheCon North America. You can submit your proposed sessions at

[GitHub] nifi pull request: Nifi 631

Github user jskora commented on the pull request: https://github.com/apache/nifi/pull/113#issuecomment-159769035 Closed by commit 226ac64ef95f3d755dfbb3d5288ba98052855473 and 4c4d62c61f7c828dbcb124090992b91d631cb22e. --- If your project is set up for it, you can reply to this email

[GitHub] nifi pull request: Nifi 631

Github user jskora closed the pull request at: https://github.com/apache/nifi/pull/113 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] nifi pull request: Fixes NIFI-1220

GitHub user gilday opened a pull request: https://github.com/apache/nifi/pull/133 Fixes NIFI-1220 `MockProcessSession` returns a new FlowFile from its `penalty` method instead of mutating then returning the given FlowFile You can merge this pull request into a Git repository by

Re: PRs

Things that make me feel better: The persistence mechanism is very similar to that of ListHDFS. https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L417 On Tue, Nov 24, 2015 at 10:56 PM,

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Github user trkurc commented on a diff in the pull request: https://github.com/apache/nifi/pull/132#discussion_r45942561 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java --- @@ -102,6 +177,94 @@

Re: absolute.path vs path for FetchFile/ListFile

It sounds like ListFile kept logic similar to GetFile which I can understand that approach. However, I do believe it makes more sense to follow the behavior of ListHDFS where the path would be absolute. Thanks Joe On Wed, Nov 25, 2015 at 1:56 PM, Tony Kurc wrote: > All, > Joe

Re: Clustered Site-toSite

Ok, that all makes sense. The main reason, we like doing it strictly as S2S is to maintain the flowfile attributes, so we would like to avoid HTTP. Otherwise we would have to rebuild some of these attributes from the content, which isn't the end of the world, but still no fun. We may consider

Re: absolute.path vs path for FetchFile/ListFile

I am reading the ListHDFS code. I can't tell if the description is wrong, the code is wrong, or I'm missing something. Description: The path is set to the absolute path of the file's directory on HDFS. For example, if the Directory property is set to /tmp then files picked up from /tmp will have

[GitHub] nifi pull request: NIFI-1192 added support for dynamic properties ...

Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/131 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [ANNOUNCE] New Apache NiFi PMC Member (and Committer) Sean Busbey

Congrats, Busbey!! On Nov 25, 2015 1:14 PM, "Tony Kurc" wrote: > On behalf of the Apache NiFI PMC, I am very pleased to announce that Sean > Busbey has accepted the PMC's invitation to become a PMC Member and > Committer on the Apache NiFi project. We greatly appreciate all of

Re: absolute.path vs path for FetchFile/ListFile

Tony, I would recommend that ListFile add both 'path' and 'absolute.path'. The 'path' would be relative to the base directory being listed. For example, if ListFile is configured to list files from /data/nifi/in and recurse subdirectories, and it finds a file named: /data/nifi/in/123/myfile.txt

Re: absolute.path vs path for FetchFile/ListFile

I certainly cannot argue with that, either. > On Nov 25, 2015, at 1:59 PM, Joe Witt wrote: > > It sounds like ListFile kept logic similar to GetFile which I can > understand that approach. > > However, I do believe it makes more sense to follow the behavior of > ListHDFS

Re: absolute.path vs path for FetchFile/ListFile

Okay, since we don't have consensus, here is what I propose: ListFile 1. absolute.path will be absolute, path will be relative to input directory FetchFile: change default property to ${absolute.path}/${filename}. Don't have a windows machine at the ready - will / work as a path separator?

Re: absolute.path vs path for FetchFile/ListFile

i'm fine with your proposal which merges mark's concept but aligns consistency of Fetch(File/HDFS) We should fix the docs for the CoreAttribute.PATH The concept of it being relative is simply too vague. We should just be honest that it is unspecified - subject to the meaning of whichever

[ANNOUNCE] New Apache NiFi PMC Member (and Committer) Sean Busbey

On behalf of the Apache NiFI PMC, I am very pleased to announce that Sean Busbey has accepted the PMC's invitation to become a PMC Member and Committer on the Apache NiFi project. We greatly appreciate all of Sean's hard work and generous contributions to the project. In addition to his

absolute.path vs path for FetchFile/ListFile

All, Joe and I commented on NIFI-631 that it didn't "just work" when wiring the processors together. ListFile was populating the attributes as described in CoreAttributes.java [1] (path being relative to the input directory, and absolute being the full path). FetchFile was using

Re: remote command execution via SSH?

I have first-cut implementation of ExecuteRemoteProcess processor at: https://github.com/xmlking/nifi-scripting/releases I tried to provide all capabilities offed by groovy-ssh (https://gradle-ssh-plugin.github.io/docs/

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Github user trkurc commented on the pull request: https://github.com/apache/nifi/pull/132#issuecomment-159806672 Another, major secondary concern is what to do with failed partial multipart puts and maybe having to do bucket cleanup. --- If your project is set up for it, you can

Re: PRs

Understood tony - thanks for digging into the review so thoroughly and Joe thank you. This is a very non-trivial contrib. On Thu, Nov 26, 2015 at 12:12 AM, Tony Kurc wrote: > I recommend we push NIFI-1107 to next release. We discovered some unfun > issues the S3 Multipart

Re: Clustered Site-toSite

The postHTTP processor has an option to send as a FlowFile to a listenHTTP processor on another NiFi. This allows you to keep the FlowFile attributes across multiple NiFis just like S2S. On Nov 25, 2015 1:58 PM, "Matthew Gaulin" wrote: > Ok, that all makes sense. The main

Re: Clustered Site-toSite

On Tue, Nov 24, 2015 at 1:38 PM, Edgardo Vega wrote: > Yeah the S2S port is set on all node. > > What should the host be set to on each machine? I first set it to the NCM > ip on each machine in the cluster. Then I set the host to be the ip of each > individual machine

Re: Clustered Site-toSite

Thank you for the info. I was working with Edgardo on this. We ended up having to set the SAME hostname on each of the source nodes, as the destination NCM uses for each of its nodes and of course open up the firewall rules so all source nodes can talk to each destination node. This seems to