[GitHub] nifi pull request: NIFI-1802 Separated /access REST API endpoints ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/376


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1803: Exclude spark classes from nifi-spar...

[alopresto]

Github user alopresto commented on the pull request:

https://github.com/apache/nifi/pull/375#issuecomment-213243013
  
Also backported to 0.x. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1803: Exclude spark classes from nifi-spar...

[alopresto]

Github user alopresto commented on the pull request:

https://github.com/apache/nifi/pull/375#issuecomment-213233268
  
Reviewed, built, no issues. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1803: Exclude spark classes from nifi-spar...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/375


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1802 Separated /access REST API endpoints ...

[alopresto]

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/376

NIFI-1802 Separated /access REST API endpoints into separate section in API 
docs.

Fixed typo in DocGenerator comments.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-1802

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/376.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #376


commit 948369bbe5fbe2bc5f81ed748472bd66ff66fb45
Author: Andy LoPresto 
Date:   2016-04-22T02:52:24Z

NIFI-1802 Separated /access REST API endpoints into separate section in API 
docs.
Fixed typo in DocGenerator comments.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1803: Exclude spark classes from nifi-spar...

[randerzander]

GitHub user randerzander opened a pull request:

https://github.com/apache/nifi/pull/375

NIFI-1803: Exclude spark classes from nifi-spark-receiver



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/randerzander/nifi master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/375.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #375


commit 18c53c5000448477a2bb46a153565a8769e085bc
Author: Randy Gelhausen 
Date:   2016-04-21T22:11:12Z

Scoped spark-streaming provided




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Where can I find documentation of implementing custom processor?

[Andy LoPresto]

Ashraf,

Are you looking for a step-by-step tutorial? The Developer Guide [1] does 
discuss Controller Services to some extent. You can also take a look at an 
example like SSLContextService and the Maven directory structure here [2].

[1] 
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#controller-services
 

[2] 
https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle
 



Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Apr 21, 2016, at 12:47 PM, Ashraf Hasson  wrote:
> 
> Hi Andy,
> 
> Can you complement your list with guides or how-tos on building a custom 
> controller service and tie that into a processor that leverages the service. 
> The only blog I found on this was Phillip Grenier's article 
> (http://www.nifi.rocks/developing-a-custom-apache-nifi-controller-service/ 
> ) 
> but it doesn't detail how the bundle directory structure and dependencies 
> were built with maven.
> Developing a Custom Apache Nifi Controller Service - NiFi ... 
> 
> www.nifi.rocks 
> How to develop a custom Apache Nifi controller service. The example in this 
> post will utilize Java Properties files.
> 
> Thanks,
> 
> 
> From: Andy LoPresto 
> Sent: April 20, 2016 6:59 PM
> To: dev@nifi.apache.org
> Subject: Re: Where can I find documentation of implementing custom processor?
> 
> Hi Keith,
> 
> I’ve moved this conversation to the dev list as that is where coding is more 
> frequently discussed as opposed to the users list. If you’re not already 
> subscribed, you can do so following the instructions here [1] to ensure you 
> see any replies.
> 
> The best approach is probably to examine the standard processors in [2]. Here 
> you will be able to see how other developers have extended the 
> AbstractProcessor class with domain-specific implementations. One example is 
> EncryptContent [3] — you can see that while a lot of the encryption logic is 
> delegated to the various *Encryptor classes, the code includes 
> processor-specific tasks like configuring PropertyDescriptors (how users 
> enter properties for the processor) and validating those values, configuring 
> relationships like success and failure, and using the session and flowfile 
> with StreamCallbacks and reporting provenance events.
> 
> For extensive documentation on writing a processor, the Developer Guide [4] 
> is the most comprehensive resource and describes the processor API, 
> documenting your processor, common patterns, and error handling & testing. If 
> you have specific questions during the development process, please feel free 
> to ask on the dev list. The community has a broad and deep knowledge and is 
> excited to help new developers with general NiFi development or very 
> domain-specific questions as well.
> 
> [1] https://nifi.apache.org/mailing_lists.html 
> 
> [2] 
> https://github.com/apache/nifi/tree/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
>  
> 
> [3] 
> https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EncryptContent.java
>  
> 
> [4] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html 
> 
> 
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Apr 20, 2016, at 3:42 PM, Keith Lim > > wrote:
>> 
>> Is there any good example of implementing a custom Processor?
>> 
>> Thanks,
>> Keith



signature.asc
Description: Message signed with OpenPGP using GPGMail

[GitHub] nifi pull request: NIFI-1554: Introducing new REST endpoints to al...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/374


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1554: Introducing new REST endpoints to al...

[bbende]

Github user bbende commented on the pull request:

https://github.com/apache/nifi/pull/374#issuecomment-213121166
  
Pulled down the latest and tested again, all looks good now, app seems to 
function properly in standalone and clustered mode.

I'm a +1 to merge this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request: MINIFI-17 Adding error handling of confi...

[JPercivall]

GitHub user JPercivall opened a pull request:

https://github.com/apache/nifi-minifi/pull/15

MINIFI-17 Adding error handling of configurations that fail to start …

…and a couple other small changes

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JPercivall/nifi-minifi MINIFI-17

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi/pull/15.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #15


commit d3792ead7ca1207980e6388b93adf42e7e7a370c
Author: Joseph Percivall 
Date:   2016-04-21T20:50:19Z

MINIFI-17 Adding error handling of configurations that fail to start and a 
couple other small changes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1554: Introducing new REST endpoints to al...

[bbende]

Github user bbende commented on the pull request:

https://github.com/apache/nifi/pull/374#issuecomment-213101837
  
I was testing a 2 node cluster locally and noticed that the part in the 
toolbar that says Connected Nodes didn't show anything after it, but in the 
cluster view it showed both nodes in the table. Not sure if this is related to 
this PR or not, still investigating.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Where can I find documentation of implementing custom processor?

[Ashraf Hasson]

Hi Andy,


Can you complement your list with guides or how-tos on building a custom 
controller service and tie that into a processor that leverages the service. 
The only blog I found on this was Phillip Grenier's article 
(http://www.nifi.rocks/developing-a-custom-apache-nifi-controller-service/) but 
it doesn't detail how the bundle directory structure and dependencies were 
built with maven.

Developing a Custom Apache Nifi Controller Service - NiFi 
...
www.nifi.rocks
How to develop a custom Apache Nifi controller service. The example in this 
post will utilize Java Properties files.



Thanks,




From: Andy LoPresto 
Sent: April 20, 2016 6:59 PM
To: dev@nifi.apache.org
Subject: Re: Where can I find documentation of implementing custom processor?

Hi Keith,

I've moved this conversation to the dev list as that is where coding is more 
frequently discussed as opposed to the users list. If you're not already 
subscribed, you can do so following the instructions here [1] to ensure you see 
any replies.

The best approach is probably to examine the standard processors in [2]. Here 
you will be able to see how other developers have extended the 
AbstractProcessor class with domain-specific implementations. One example is 
EncryptContent [3] - you can see that while a lot of the encryption logic is 
delegated to the various *Encryptor classes, the code includes 
processor-specific tasks like configuring PropertyDescriptors (how users enter 
properties for the processor) and validating those values, configuring 
relationships like success and failure, and using the session and flowfile with 
StreamCallbacks and reporting provenance events.

For extensive documentation on writing a processor, the Developer Guide [4] is 
the most comprehensive resource and describes the processor API, documenting 
your processor, common patterns, and error handling & testing. If you have 
specific questions during the development process, please feel free to ask on 
the dev list. The community has a broad and deep knowledge and is excited to 
help new developers with general NiFi development or very domain-specific 
questions as well.

[1] https://nifi.apache.org/mailing_lists.html
[2] 
https://github.com/apache/nifi/tree/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
[3] 
https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EncryptContent.java
[4] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Apr 20, 2016, at 3:42 PM, Keith Lim 
mailto:keith@ds-iq.com>> wrote:

Is there any good example of implementing a custom Processor?

Thanks,
Keith

[GitHub] nifi pull request: NIFI-1554: Introducing new REST endpoints to al...

[bbende]

Github user bbende commented on the pull request:

https://github.com/apache/nifi/pull/374#issuecomment-213080400
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1661 add random() function to expression l...

[jskora]

Github user jskora commented on the pull request:

https://github.com/apache/nifi/pull/322#issuecomment-213070365
  
@ckmcd this has been merged to the 1.0 and 0.x branches.  Looks great and 
works well.

I forgot to put  "this closes" on the commit, could you please close the PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1554: Introducing new REST endpoints to al...

[mcgilman]

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/374

NIFI-1554: Introducing new REST endpoints to align with the authorizable 
resources

- Introducing new REST endpoints to align with the authorizable resources.
- Additional changes to support new endpoints.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-1554

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/374.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #374


commit 1c159ced404c5e931e829a8eb54d8c391b56823f
Author: Matt Gilman 
Date:   2016-04-21T18:47:10Z

NIFI-1554:
- Introducing new REST endpoints to align with the authorizable resources.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-361 - Create Processors to mutate JSON dat...

[YolandaMDavis]

Github user YolandaMDavis commented on a diff in the pull request:

https://github.com/apache/nifi/pull/354#discussion_r60631450
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TransformJSON.java
 ---
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.util.StopWatch;
+
+import com.bazaarvoice.jolt.Shiftr;
+import com.bazaarvoice.jolt.Removr;
+import com.bazaarvoice.jolt.Chainr;
+import com.bazaarvoice.jolt.Defaultr;
+import com.bazaarvoice.jolt.Transform;
+import com.bazaarvoice.jolt.JsonUtils;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"json", "jolt", "transform", "shiftr", "chainr", "defaultr", 
"removr"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Applies a list of JOLT specifications to the 
flowfile JSON payload. A new FlowFile is created "
++ "with transformed content and is routed to the 'success' 
relationship. If the JSON transform "
++ "fails, the original FlowFile is routed to the 'failure' 
relationship")
+public class TransformJSON extends AbstractProcessor {
+
+public static final AllowableValue SHIFTR = new 
AllowableValue("Shift", "Shift Transform DSL", "This JOLT transformation will 
shift input JSON/data to create the output JSON/data.");
+public static final AllowableValue CHAINR = new 
AllowableValue("Chain", "Chain Transform DSL", "Execute list of JOLT 
transformations.");
+public static final AllowableValue DEFAULTR = new 
AllowableValue("Default", "Default Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+public static final AllowableValue REMOVR = new 
AllowableValue("Remove", "Remove Transform DSL", " This JOLT transformation 
will apply default values to the output JSON/data.");
+
+public static final PropertyDescriptor JOLT_SPEC = new 
PropertyDescriptor.Builder()
+.name("Jolt Specification")
+.description("Jolt Specification for transform of JSON data.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.addValidator(new JOLTSpecValidator())
+.required(true)
+.build();
+
+public static final PropertyDescriptor JOLT_TRANSF

Re: NiFi Filenames too long

[shenya]

Please try long path tool.



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Filenames-too-long-tp1096p9437.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...

[olegz]

Github user olegz commented on the pull request:

https://github.com/apache/nifi/pull/347#issuecomment-213020240
  
@pvillard31 Actually I totally forgot about the effort I started a while 
back. There is actually a branch for it 
https://github.com/olegz/nifi/tree/int-test/nifi-integration-tests. Not much 
there but S2S test is (totally headless). 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...

[olegz]

Github user olegz commented on the pull request:

https://github.com/apache/nifi/pull/347#issuecomment-212995061
  
@pvillard31 let me dig some more. Yes you have to have the "other side" 
running but I had some other code bits to simulate that. I'll post here once I 
find


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [Discuss] Expression Language Options to Add Decimal Support

[Joe Percivall]

Mark,


After thinking through this a bit more I believe you're right. We can assume 
this logic for every operation:

Two whole numbers as input returns a whole number
A whole number and a decimal as input returns a decimal
Two decimals as input returns a Decimal

I believe, one problem with your explanation as written is "...infer the return 
type and allow the user to explicitly change it via toNumber() and toDecimal() 
calls if need be". More specifically you would need to call toNumber/toDecimal 
on the inputs in order to change the return type. You can't change the return 
type after the expression runs (without the potential loss of 
information/precision of converting between the two) because once the 
expression evaluator exits it needs to have already interpreted it as a long or 
double which would occur before the user is able to call toNumber/toDecimal. So 
the user would have to control the return type would by calling 
toNumber/toDecimal on the inputs. 

That leads to the one confusing bit about this approach, when a user attempts 
to do an operation on two whole numbers that they would expect to return a 
decimal (like 8/5 = 1.6). In order to get their expected output the user would 
have to know to explicitly convert at least one of them to a double by using 
the toDecimal operation. 

That being said it's probably that's worth it to keep down the verbosity and 
still have backwards compatibility. 
 

Joe
- - - - - - Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com



On Thursday, April 21, 2016 10:20 AM, Mark Payne  wrote:



Joe,

I am definitely in favor of #3. I think if we perform some sort of binary 
operation on two numbers,
we should return a Decimal type if either of the operands is a Decimal and a 
Number (Long) type
if both operands are Longs. We would also need a toDecimal() method that would 
convert a Number
type ot a Decimal type and we would need to support calling toNumber() on a 
decimal as well.

Given this, though, I think it makes sense to infer the return type and allow 
the user to explicitly
change it via toNumber() and toDecimal() calls if need be. With the addition of 
these functions, I don't
think it would be 'shady' to interpret the types at all. We already do 
interpret types in many cases, for
instance when we have an expression such as ${num1:divide(${num2}) } -- in this 
case, num1 and num2
are attributes, so they are strings. We already implicitly convert them to 
numbers. Going forward, we should
convert them to either Number or Decimal type based on the regex that they 
match. I believe this also
addresses the concern of not being able to override. I don't think backward 
compatibility is an issue here
either, as we currently would just fail and throw an Exception if attempting to 
divide a string like "2.3"

Number 3 seems like a slam-dunk to me in terms of pros vs cons.

Thanks
-Mark



> On Apr 20, 2016, at 10:16 PM, Joe Percivall  
> wrote:
> 
> Hello Dev list,
> I've been working on a couple improvements that deal with utilizing 
> expression language to do data analysis and this has exposed a couple issues 
> with the typing of numbers in Expression Language (EL). The improvements are 
> a continuation of the topic I presented on at the Apache NiFi MeetUp group in 
> MD[1]. 
> The primary issue I came across is that currently in EL all numbers 
> interpreted as longs[2], which only store whole numbers. This leads to 
> problems when trying to do things like dividing whole numbers or just trying 
> to add/subtract decimals. I am actually surprised that the request for 
> decimals this hasn't come up before. That being said, after some initial 
> discussion with Tony and Joe, I believe that there are four potential ways 
> forward.
> 1: Create a new EL type "decimal" backed by a double[3] and new methods to 
> support it ("add_decimal"): This allows the user to explicitly choose whether 
> or not they want to use decimal or whole numbers. It retains the simple 
> use-cases that use whole numbers while opening up the new use-cases using 
> decimals. One down side is that it is more verbose. It means adding a new 
> function for each math operation. This is backwards compatible.
> 
> 2: Back all numbers by doubles instead of longs: The easy to implement and 
> retains very concise methods ("add", "subtract", etc..). A few cons, doubles 
> have a lower precision than longs[4],  can lead to rounding errors[5] and 
> could be confusing for users who only work with whole numbers to suddenly 
> find decimals.This is not backwards compatible.
> 3: Create a new EL type "decimal" back by a double[3] and attempt to smartly 
> automatically cast depending on method/input/output: This would allow for the 
> positives of having decimals and whole numbers in addition to having concise 
> methods. The main cons being a much longer implementation time to do it right 
> and the "shadiness" of doing things automatically for the user. Also this 
> wou

Issues with re-starting NiFI after modifying a custom processor

[idioma]

Hi,
I am running NiFi on WIndows simply by clicking on the .bat file. It loads
all the files and I can access to localhost:8080/nifi and see out-of-the-box
processors and custom ones (if any). However, if I make a change to an
existing custom processor and I want to stop and re-start NiFi, this does
not seem to work properly. I simply shut down the Dos Windows and when I
restart the .bat file it shows as still running. Changes to the processor
are not reflected and I need to actually re-start my Windows to see some
improvements. 

What is happening? Am I wrongly stopping/starting/restarting NiFi. 

Thank you



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Issues-with-re-starting-NiFI-after-modifying-a-custom-processor-tp9449.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: [Discuss] Expression Language Options to Add Decimal Support

[Mark Payne]

Joe,

I am definitely in favor of #3. I think if we perform some sort of binary 
operation on two numbers,
we should return a Decimal type if either of the operands is a Decimal and a 
Number (Long) type
if both operands are Longs. We would also need a toDecimal() method that would 
convert a Number
type ot a Decimal type and we would need to support calling toNumber() on a 
decimal as well.

Given this, though, I think it makes sense to infer the return type and allow 
the user to explicitly
change it via toNumber() and toDecimal() calls if need be. With the addition of 
these functions, I don't
think it would be 'shady' to interpret the types at all. We already do 
interpret types in many cases, for
instance when we have an expression such as ${num1:divide(${num2}) } -- in this 
case, num1 and num2
are attributes, so they are strings. We already implicitly convert them to 
numbers. Going forward, we should
convert them to either Number or Decimal type based on the regex that they 
match. I believe this also
addresses the concern of not being able to override. I don't think backward 
compatibility is an issue here
either, as we currently would just fail and throw an Exception if attempting to 
divide a string like "2.3"

Number 3 seems like a slam-dunk to me in terms of pros vs cons.

Thanks
-Mark


> On Apr 20, 2016, at 10:16 PM, Joe Percivall  
> wrote:
> 
> Hello Dev list,
> I've been working on a couple improvements that deal with utilizing 
> expression language to do data analysis and this has exposed a couple issues 
> with the typing of numbers in Expression Language (EL). The improvements are 
> a continuation of the topic I presented on at the Apache NiFi MeetUp group in 
> MD[1]. 
> The primary issue I came across is that currently in EL all numbers 
> interpreted as longs[2], which only store whole numbers. This leads to 
> problems when trying to do things like dividing whole numbers or just trying 
> to add/subtract decimals. I am actually surprised that the request for 
> decimals this hasn't come up before. That being said, after some initial 
> discussion with Tony and Joe, I believe that there are four potential ways 
> forward.
> 1: Create a new EL type "decimal" backed by a double[3] and new methods to 
> support it ("add_decimal"): This allows the user to explicitly choose whether 
> or not they want to use decimal or whole numbers. It retains the simple 
> use-cases that use whole numbers while opening up the new use-cases using 
> decimals. One down side is that it is more verbose. It means adding a new 
> function for each math operation. This is backwards compatible.
> 
> 2: Back all numbers by doubles instead of longs: The easy to implement and 
> retains very concise methods ("add", "subtract", etc..). A few cons, doubles 
> have a lower precision than longs[4],  can lead to rounding errors[5] and 
> could be confusing for users who only work with whole numbers to suddenly 
> find decimals.This is not backwards compatible.
> 3: Create a new EL type "decimal" back by a double[3] and attempt to smartly 
> automatically cast depending on method/input/output: This would allow for the 
> positives of having decimals and whole numbers in addition to having concise 
> methods. The main cons being a much longer implementation time to do it right 
> and the "shadiness" of doing things automatically for the user. Also this 
> would mean the user wouldn't have the option to explicitly override  This is 
> not backwards compatible.
> 4: Create a new EL type "decimal" backed by a double[3] and overload the 
> existing methods with an additional parameter to specify return type to 
> support it: This would allow for the positives of having decimals and whole 
> numbers in addition to having concise method names but this may cause 
> confusion with less technical users who aren't used to specifying return 
> types. This is backwards compatible.
> 
> The options that are not backwards compatible would need to wait to be 
> implemented until 1.0.
> The current option I am leaning towards is number 1 due to the explicitness 
> and greater control it gives the user. While it is more verbose I think the 
> decimal vs whole number syntax will be easy for even non-technical users to 
> pick up. Also I currently have a PR for it up here[6].
> Any other ideas or suggestions are welcome!
> [1] http://www.meetup.com/ApacheNiFi/events/229158777/
> [2] https://docs.oracle.com/javase/7/docs/api/java/lang/Long.html
> [3] https://docs.oracle.com/javase/7/docs/api/java/lang/Double.html
> [4] https://docs.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html
> [5] http://stackoverflow.com/questions/960072/rounding-errors
> [6] https://issues.apache.org/jira/browse/NIFI-1662
> Joe- - - - - - Joseph Percivalllinkedin.com/in/Percivalle: 
> joeperciv...@yahoo.com

Re: [Discuss] Expression Language Options to Add Decimal Support

[Pierre Villard]

Hi Joe,

In my opinion, I think I would feel more "natural" to go with option 4.
Option 1 is fine as well but I think that adding a lot of functions could
be confusing with the auto-completion feature.


2016-04-21 4:47 GMT+02:00 Andy LoPresto :

> I would support option 4, then option 1. I think this is a significant
> change, so backward-compatibility is important, and I don’t see specifying
> the return type to be a high hurdle provided the documentation is clear. If
> someone is writing in expression language, there is a minimum expectation
> of exposure to the documentation, as it is a bespoke implementation, not an
> OTS offering that they would have previous experience with.
>
> I think options 2 and 3 introduce the opportunity to compromise
> previously-functioning expressions in a way that could be very difficult to
> troubleshoot.
>
> I do appreciate you discovering and diagnosing this. I can imagine there
> are already scenarios where EL is failing silently and this should
> definitely be fixed correctly.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Apr 20, 2016, at 7:16 PM, Joe Percivall  > wrote:
>
> Hello Dev list,
> I've been working on a couple improvements that deal with utilizing
> expression language to do data analysis and this has exposed a couple
> issues with the typing of numbers in Expression Language (EL). The
> improvements are a continuation of the topic I presented on at the Apache
> NiFi MeetUp group in MD[1].
> The primary issue I came across is that currently in EL all numbers
> interpreted as longs[2], which only store whole numbers. This leads to
> problems when trying to do things like dividing whole numbers or just
> trying to add/subtract decimals. I am actually surprised that the request
> for decimals this hasn't come up before. That being said, after some
> initial discussion with Tony and Joe, I believe that there are four
> potential ways forward.
> 1: Create a new EL type "decimal" backed by a double[3] and new methods to
> support it ("add_decimal"): This allows the user to explicitly choose
> whether or not they want to use decimal or whole numbers. It retains the
> simple use-cases that use whole numbers while opening up the new use-cases
> using decimals. One down side is that it is more verbose. It means adding a
> new function for each math operation. This is backwards compatible.
>
> 2: Back all numbers by doubles instead of longs: The easy to implement and
> retains very concise methods ("add", "subtract", etc..). A few cons,
> doubles have a lower precision than longs[4],  can lead to rounding
> errors[5] and could be confusing for users who only work with whole numbers
> to suddenly find decimals.This is not backwards compatible.
> 3: Create a new EL type "decimal" back by a double[3] and attempt to
> smartly automatically cast depending on method/input/output: This would
> allow for the positives of having decimals and whole numbers in addition to
> having concise methods. The main cons being a much longer implementation
> time to do it right and the "shadiness" of doing things automatically for
> the user. Also this would mean the user wouldn't have the option to
> explicitly override  This is not backwards compatible.
> 4: Create a new EL type "decimal" backed by a double[3] and overload the
> existing methods with an additional parameter to specify return type to
> support it: This would allow for the positives of having decimals and whole
> numbers in addition to having concise method names but this may cause
> confusion with less technical users who aren't used to specifying return
> types. This is backwards compatible.
>
> The options that are not backwards compatible would need to wait to be
> implemented until 1.0.
> The current option I am leaning towards is number 1 due to the
> explicitness and greater control it gives the user. While it is more
> verbose I think the decimal vs whole number syntax will be easy for even
> non-technical users to pick up. Also I currently have a PR for it up
> here[6].
> Any other ideas or suggestions are welcome!
> [1] http://www.meetup.com/ApacheNiFi/events/229158777/
> [2] https://docs.oracle.com/javase/7/docs/api/java/lang/Long.html
> [3] https://docs.oracle.com/javase/7/docs/api/java/lang/Double.html
> [4]
> https://docs.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html
> [5] http://stackoverflow.com/questions/960072/rounding-errors
> [6] https://issues.apache.org/jira/browse/NIFI-1662
> Joe- - - - - - Joseph Percivalllinkedin.com/in/Percivalle:
>  joeperciv...@yahoo.com
>
>
>

Re: [DISCUSS] From Contributor to Committer

[Aldrin Piri]

Looks like a good set of information and listing of capacities in which the
community can contribute for progression.

On Wed, Apr 20, 2016 at 6:23 PM, Joe Witt  wrote:

> Bryan,
>
> That is a good point and I do think we should avoid putting such a
> rule/requirement in place.  However, practically speaking I would
> imagine that is how it will play out most often.
>
> I've taken the comments of this thread largely based on Tony's
> excellent start and created a draft wiki page for it here [1].  If
> discussion remains on track then I'll assume lazy consensus and remove
> the draft notice.
>
> [1]
> https://cwiki.apache.org/confluence/display/NIFI/Progression+from+user+to+Project+Management+Committee
>
> Thanks
> Joe
>
> On Wed, Apr 20, 2016 at 12:05 PM, Bryan Bende  wrote:
> > So are we saying as a community that a contributor has to first become a
> > committer, and then only after continued consistent engagement could then
> > be considered for PMC?
> >
> > I don't have any issue with that approach, although it is not exactly
> what
> > I thought when we first created the two tiers.
> >
> > On Wed, Apr 20, 2016 at 11:10 AM, Joe Witt  wrote:
> >
> >> Tony,
> >>
> >> There appears to be consensus around these thoughts.  Perhaps we
> >> should document this on a Wiki page?
> >>
> >> I think generally for committer status it would be good to see a
> >> number of these things for a period of time and then for PMC status to
> >> see those contributions continue and ideally expand for a longer
> >> duration.  Another few months?
> >>
> >> Thanks
> >> Joe
> >>
> >> On Wed, Apr 13, 2016 at 2:58 PM, Joe Witt  wrote:
> >> > Tony,
> >> >
> >> > I agree with the points you raise and the completeness of the
> >> > perspective you share. I do think we should add to that a focus on
> >> > licensing and legal aspects.
> >> >
> >> > - The individual has shown an effort to aid the community in producing
> >> > release which are consistent with ASF licensing requirements and the
> >> > guidance followed in the Apache NiFi community to adhere to those
> >> > policies.  This understanding could be shown when introducing new
> >> > dependencies (including transitive) by ensuring that all licensing and
> >> > notice updates have occurred.  Another good example is flagging
> >> > potentially copyrighted or insufficiently cited items like Skora found
> >> > recently in the Kafka tests.  One of our most important jobs as a
> >> > community is to put out legal releases and that is certainly a team
> >> > effort!
> >> >
> >> > Thanks
> >> > Joe
> >> >
> >> > On Sun, Apr 10, 2016 at 10:56 PM, Sean Busbey 
> wrote:
> >> >> Thanks for starting this Tony!
> >> >>
> >> >> As a PMC member, I really try to focus on things that help the
> >> >> community where we tend to have limited bandwidth: reviews weigh
> >> >> heavily, as does helping out new folks on users@, and doing public
> >> >> talking/workshops.
> >> >>
> >> >> I also am inclined to vote in favor of folks who show the kind of
> >> >> project dedication that we expect from PMC members. While we still
> >> >> need to do a better job of describing those things, at the moment I'm
> >> >> thinking of things like voting on release candidates, watching out
> for
> >> >> our trademarks, and investing the time needed to handle our licensing
> >> >> responsibilities.
> >> >>
> >> >> On Sun, Apr 10, 2016 at 9:38 AM, Tony Kurc  wrote:
> >> >>> First off, I recommend this reading this page to understand what the
> >> Apache
> >> >>> NiFi PMC draws from when making a decision
> >> >>>
> >> >>> http://community.apache.org/contributors/index.html
> >> >>>
> >> >>> I thought it would be helpful for me to walk through how I interpret
> >> that
> >> >>> guidance, and what that means for NiFi. For those that didn't read,
> >> there
> >> >>> are four aspects of contribution that are worth considering someone
> for
> >> >>> committership: community, project, documentation and code. Really,
> the
> >> >>> committer decision comes down to: has this person built up enough
> >> merit in
> >> >>> the community that I have a high degree of confidence that I trust
> >> him/her
> >> >>> with write access to the code and website.
> >> >>>
> >> >>> Given that merit and trust are subjective measures, how does the PMC
> >> make
> >> >>> those decisions? We, the PMC, have attempted to make this as
> >> evidence-based
> >> >>> as possible. When discussing a contributor for being considered for
> >> >>> committer access, we attempt to put together a corpus of interaction
> >> in the
> >> >>> community, both negative and positive, and use this as a basis for
> >> >>> discussion. The interaction with the community can include:
> >> >>>
> >> >>> - Interaction on the mailing lists - is this person helping others?
> Is
> >> this
> >> >>> person using the community to enhance his/her understanding of the
> >> project
> >> >>> or the apache foundation?
> >> >>> - Code contributions - is this person contributing code th

Re: Clarifications/Suggestions on Using NIFI.

[Bryan Bende]

Hello,

1) Any processor that brings data into NiFi will store it in the NiFi's
internal repositories. If this wasn't done then the data could not be
reliably delivered, if the data were only in memory and NiFi crashed then
NiFi would have no way of knowing what has been delivered and what hasn't.

2) Currently users have to share a single instance, but the community is
working towards multi-tenancy [1].

3) The SMTP hostname is the hostname of the mail server that is going to
send your email, so if you are sending from your GMail account it would be
GMail's SMTP server. There are some services out there for sending emails
like Mailgun [2].

-Bryan

[1] https://cwiki.apache.org/confluence/display/NIFI/Multi-Tentant+Dataflow
[2] http://www.mailgun.com/


On Thu, Apr 21, 2016 at 8:40 AM, Balaji K Hari 
wrote:

> Hi Joe/Team,
>
>
>
> As per your suggestion, we have one specific use case to be developed
> using NIFI and need your assistance on this requirement to be achieved.
>
>
>
> 1) We are trying to pull the data from multiple RDBMS databases and
> putting in to Amazon S3 using NIFI. When we have done this, we found that
> the database data would be put into NIFI local first and then put the same
> into S3.
>
>
>
> However we want to know the direct way, where the data would directly
> pulled from database and put into S3. So could you please suggest any
> processor where we can achieve this. Please find the diagram attached.
>
>
>
> 2) Also when we configure the NIFI, we observed that it is a Single
> User instance i.e. only 1 user can work on NIFI and if any other users
> tries to do anything it will reflects in the initial user’s instance. Can
> NIFI be configured as a multi-user instance where all the users can work on
> it at a time?
>
>
>
> 3) I am trying to email notifications using “PutEmail” processor
> Please explain the parameters to be put in the “PutEmail” Processor. The
> first parameter it asks is “SMTP hostname”. If I want to send email from my
> gmail id to yahoo id. Should I put SMTP hostname of gmail or Yahoo ? Also
> where can I get details of SMTP hostnames and port number if this processor
> is been executed from the client network?
>
>
>
> Looking forward for your valuable inputs/suggestions.
>
>
>
> Regards,
>
> ___
>
> [image: Email_CBE.gif]*Balaji KNV_Hari*
>
> Technical Architect
>
>
>
> *From:* Joe Witt [mailto:joe.w...@gmail.com]
> *Sent:* Thursday, April 14, 2016 7:40 PM
> *To:* dev@nifi.apache.org; Balaji K Hari
> *Subject:* Re: Clarifications/Suggestions on Using NIFI.
>
>
>
>
>
>
>
> On Thu, Apr 14, 2016 at 10:09 AM, Joe Witt  wrote:
>
> Hello
>
>
>
> You received a near immediate response which you can find here [1].  But
> since you are not subscribed to the mailing list you did not receive the
> response.  Please subscribe here [2].
>
>
>
> [1]
> http://mail-archives.apache.org/mod_mbox/nifi-dev/201604.mbox/%3CCALJK9a6h0qv14%3DyqN4mqRDJQYaimwO1V1DviqFntt0Xy2dTYQQ%40mail.gmail.com%3E
>
> [2] https://nifi.apache.org/mailing_lists.html
>
>
>
> Thanks
>
> Joe
>
>
>
> On Thu, Apr 14, 2016 at 10:03 AM, Balaji K Hari 
> wrote:
>
> Hi Dev Team,
>
>
>
> Good Morning!!
>
>
>
> I hope you are trying hard by taking time from your busy schedule to
> provide the inputs for the below questions.
>
>
>
> It would be really helpful, if you can answer these at earliest as based
> on this we want to try with other options to achieve the functionalities as
> per requirements.
>
>
>
> Thanks a lot for your assistance… Looking forward for your reply.
>
>
>
> Regards,
>
> ___
>
> [image: Email_CBE.gif]*Balaji KNV_Hari*
>
> Technical Architect
>
>
>
> *From:* Balaji K Hari
> *Sent:* Tuesday, April 12, 2016 7:57 PM
> *To:* 'd...@nifi.incubator.apache.org'
> *Subject:* Clarifications/Suggestions on Using NIFI.
> *Importance:* High
>
>
>
> Hi Team,
>
>
>
> Based on the project requirements, I was looking at different features
> included in Apache NIFI and found that this would be the good way to
> interact with the Development team who have developed NIFI and are looking
> for suggestions/inputs from the User community to improvise the product and
> also it is a great medium where the users who are using this NIFI would get
> the valuable inputs from the developers for their requirements.
>
>
>
> Need your assistance/inputs on the below requirements and how these can be
> implemented in NIFI to achieve the solution.
>
>
>
> è  I have observed that, *Event Based Scheduling/Any Trigger Based
> Scheduling* is yet to be included in the latest NIFI product. Any
> workarounds/alternatives to achieve this?
>
> è  Can *Spark/Hive Jobs* can be scheduled on time basis and also executed
> through NIFI? If Yes, please suggest how can we do this?
>
> è  Can we get the data from multiple tables of *Oracle/SQL
> Server/Teradata* and put directly in *S3/HDFS* and also 

Clarifications/Suggestions on Using NIFI.

[Balaji K Hari]

Hi Joe/Team,

As per your suggestion, we have one specific use case to be developed using 
NIFI and need your assistance on this requirement to be achieved.


1) We are trying to pull the data from multiple RDBMS databases and  
putting in to Amazon S3 using NIFI. When we have done this, we found that the 
database data would be put into NIFI local first and then put the same into S3.

However we want to know the direct way, where the data would directly pulled 
from database and put into S3. So could you please suggest any processor where 
we can achieve this. Please find the diagram attached.


2) Also when we configure the NIFI, we observed that it is a Single User 
instance i.e. only 1 user can work on NIFI and if any other users tries to do 
anything it will reflects in the initial user’s instance. Can NIFI be 
configured as a multi-user instance where all the users can work on it at a 
time?



3) I am trying to email notifications using “PutEmail” processor Please 
explain the parameters to be put in the “PutEmail” Processor. The first 
parameter it asks is “SMTP hostname”. If I want to send email from my gmail id 
to yahoo id. Should I put SMTP hostname of gmail or Yahoo ? Also where can I 
get details of SMTP hostnames and port number if this processor is been 
executed from the client network?

Looking forward for your valuable inputs/suggestions.

Regards,
___
[Email_CBE.gif]Balaji KNV_Hari
Technical Architect

From: Joe Witt [mailto:joe.w...@gmail.com]
Sent: Thursday, April 14, 2016 7:40 PM
To: dev@nifi.apache.org; Balaji K Hari
Subject: Re: Clarifications/Suggestions on Using NIFI.



On Thu, Apr 14, 2016 at 10:09 AM, Joe Witt 
mailto:joe.w...@gmail.com>> wrote:
Hello

You received a near immediate response which you can find here [1].  But since 
you are not subscribed to the mailing list you did not receive the response.  
Please subscribe here [2].

[1] 
http://mail-archives.apache.org/mod_mbox/nifi-dev/201604.mbox/%3CCALJK9a6h0qv14%3DyqN4mqRDJQYaimwO1V1DviqFntt0Xy2dTYQQ%40mail.gmail.com%3E
[2] https://nifi.apache.org/mailing_lists.html

Thanks
Joe

On Thu, Apr 14, 2016 at 10:03 AM, Balaji K Hari 
mailto:balaji.h...@capgemini.com>> wrote:
Hi Dev Team,

Good Morning!!

I hope you are trying hard by taking time from your busy schedule to provide 
the inputs for the below questions.

It would be really helpful, if you can answer these at earliest as based on 
this we want to try with other options to achieve the functionalities as per 
requirements.

Thanks a lot for your assistance… Looking forward for your reply.

Regards,
___
[Email_CBE.gif]Balaji KNV_Hari
Technical Architect

From: Balaji K Hari
Sent: Tuesday, April 12, 2016 7:57 PM
To: 'd...@nifi.incubator.apache.org'
Subject: Clarifications/Suggestions on Using NIFI.
Importance: High

Hi Team,

Based on the project requirements, I was looking at different features included 
in Apache NIFI and found that this would be the good way to interact with the 
Development team who have developed NIFI and are looking for suggestions/inputs 
from the User community to improvise the product and also it is a great medium 
where the users who are using this NIFI would get the valuable inputs from the 
developers for their requirements.

Need your assistance/inputs on the below requirements and how these can be 
implemented in NIFI to achieve the solution.


==>  I have observed that, Event Based Scheduling/Any Trigger Based Scheduling 
is yet to be included in the latest NIFI product. Any workarounds/alternatives 
to achieve this?

==>  Can Spark/Hive Jobs can be scheduled on time basis and also executed 
through NIFI? If Yes, please suggest how can we do this?

==>  Can we get the data from multiple tables of Oracle/SQL Server/Teradata and 
put directly in S3/HDFS and also directly to RedShift/Any database? If Yes, 
please suggest how can we do this?

==>  Also can we do the transformations/manipulations on the data while moving 
it to S3/HDFS from RDBMS databases? If Yes, please suggest how can we do this?

==>  Can we do the validations and also find the duplicate data/records before 
you put the data into S3/HDFS. For example, I have moved the data from RDBMS 
tables into S3 and as part of daily loads, I need to check whether any 
duplicate records are present in the new load and need to remove those records 
while data movement itself. Please provide your inputs how can we do this?

==>  Also can you provide valuable inputs on how can we achieve the workflow 
execution  dependency i.e. For example, I have designed one workflow and based 
on this 1st workflow execution completion, I need to start the second workflow 
else need to start another workflow. Can this be achieved in NIFI?

It would be really helpful and appreciated on the above inputs, as you would be 

Re: Reg: Tailing a remote file

[Pierre Villard]

Sourav,

If what is behind your '*' has some format logic (like dates), you could
try using expression language.
Otherwise there is an open JIRA for this: NIFI-1170.



2016-04-21 14:42 GMT+02:00 Sourav Gulati :

> Simon,
>
> I see an issue with tail processor .
>
> I have to give the exact name of the file to tail. I can give something
> like /home/user/abc*.log
>
> Is there anyway I can do this?
>
> Regards,
> Sourav Gulati
>
> -Original Message-
> From: Simon Ball [mailto:sb...@hortonworks.com]
> Sent: Thursday, April 21, 2016 3:40 PM
> To: dev@nifi.apache.org
> Subject: Re: Reg: Tailing a remote file
>
> If you're using a rolling file name pattern you will want to tail the
> current log file, and essentially ignore the rolled history, unless you
> need to represses or miss data.
>
> TailFile is a local processor, so you would run a nifi agent on the box
> that has the files on, use tail file -> remote process group -> input port
> on core nifi instance -> wherever your data is going next.
>
> The local/remote setting comes up when clustering nifi and using something
> like an NFS mount to get to the files being tailed, which is a different
> way of doing this. The remote will lead to state about where you've gotten
> to being stored differently to support the cluster continuing to tail from
> alternative nifi cluster nodes.
>
> Simon
>
>
> > On 21 Apr 2016, at 11:38, Sourav Gulati 
> wrote:
> >
> > Hi Simon/Team.
> >
> > Please provide some good example of tail processor.
> >
> > I need some example for " Rolling Filename Pattern".
> >
> > Also, as per my knowledge TailFile processor is used for tailing local
> file then what does location attribute signifies, if I set it to remote.
> >
> > Regards,
> > Sourav Gulati
> >
> > -Original Message-
> > From: Simon Ball [mailto:sb...@hortonworks.com]
> > Sent: Wednesday, April 20, 2016 2:27 PM
> > To: dev@nifi.apache.org
> > Subject: Re: Reg: Tailing a remote file
> >
> > Site-to-site is in pretty much every version of nifi released under
> apache.
> >
> >> On 20 Apr 2016, at 10:48, Sourav Gulati 
> wrote:
> >>
> >> Thanks Simon. Which version of Nifi support site to site?
> >>
> >> Regards,
> >> Sourav Gulati
> >>
> >> -Original Message-
> >> From: Simon Ball [mailto:sb...@hortonworks.com]
> >> Sent: Wednesday, April 20, 2016 1:28 PM
> >> To: dev@nifi.apache.org
> >> Subject: Re: Reg: Tailing a remote file
> >>
> >> The best way to do this would be to run a nifi agent on the remote
> system, using the TailFile processor, and then use site-to-site to get that
> into your core nifi.
> >>
> >> Simon
> >>
> >>
> >>> On 20 Apr 2016, at 09:49, Sourav Gulati 
> wrote:
> >>>
> >>> Hi Team,
> >>>
> >>> We want to process logs from a file which is getting updated
> continuously on remote system.
> >>>
> >>> What is the best way to do that in NiFi?
> >>>
> >>> Regards,
> >>> Sourav Gulati
> >>>
> >>>
> >>> 
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
> >>
> >> 
> >>
> >>
> >>
> >>
> >>
> >>
> >> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
> >
> >
> > 
> >
> >
> >
> >
> >
> >
> > NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
> >
>
> 
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited w

RE: Reg: Tailing a remote file

[Sourav Gulati]

Simon,

I see an issue with tail processor .

I have to give the exact name of the file to tail. I can give something like 
/home/user/abc*.log

Is there anyway I can do this?

Regards,
Sourav Gulati

-Original Message-
From: Simon Ball [mailto:sb...@hortonworks.com]
Sent: Thursday, April 21, 2016 3:40 PM
To: dev@nifi.apache.org
Subject: Re: Reg: Tailing a remote file

If you're using a rolling file name pattern you will want to tail the current 
log file, and essentially ignore the rolled history, unless you need to 
represses or miss data.

TailFile is a local processor, so you would run a nifi agent on the box that 
has the files on, use tail file -> remote process group -> input port on core 
nifi instance -> wherever your data is going next.

The local/remote setting comes up when clustering nifi and using something like 
an NFS mount to get to the files being tailed, which is a different way of 
doing this. The remote will lead to state about where you've gotten to being 
stored differently to support the cluster continuing to tail from alternative 
nifi cluster nodes.

Simon


> On 21 Apr 2016, at 11:38, Sourav Gulati  wrote:
>
> Hi Simon/Team.
>
> Please provide some good example of tail processor.
>
> I need some example for " Rolling Filename Pattern".
>
> Also, as per my knowledge TailFile processor is used for tailing local file 
> then what does location attribute signifies, if I set it to remote.
>
> Regards,
> Sourav Gulati
>
> -Original Message-
> From: Simon Ball [mailto:sb...@hortonworks.com]
> Sent: Wednesday, April 20, 2016 2:27 PM
> To: dev@nifi.apache.org
> Subject: Re: Reg: Tailing a remote file
>
> Site-to-site is in pretty much every version of nifi released under apache.
>
>> On 20 Apr 2016, at 10:48, Sourav Gulati  wrote:
>>
>> Thanks Simon. Which version of Nifi support site to site?
>>
>> Regards,
>> Sourav Gulati
>>
>> -Original Message-
>> From: Simon Ball [mailto:sb...@hortonworks.com]
>> Sent: Wednesday, April 20, 2016 1:28 PM
>> To: dev@nifi.apache.org
>> Subject: Re: Reg: Tailing a remote file
>>
>> The best way to do this would be to run a nifi agent on the remote system, 
>> using the TailFile processor, and then use site-to-site to get that into 
>> your core nifi.
>>
>> Simon
>>
>>
>>> On 20 Apr 2016, at 09:49, Sourav Gulati  wrote:
>>>
>>> Hi Team,
>>>
>>> We want to process logs from a file which is getting updated continuously 
>>> on remote system.
>>>
>>> What is the best way to do that in NiFi?
>>>
>>> Regards,
>>> Sourav Gulati
>>>
>>>
>>> 
>>>
>>>
>>>
>>>
>>>
>>>
>>> NOTE: This message may contain information that is confidential, 
>>> proprietary, privileged or otherwise protected by law. The message is 
>>> intended solely for the named addressee. If received in error, please 
>>> destroy and notify the sender. Any use of this email is prohibited when 
>>> received in error. Impetus does not represent, warrant and/or guarantee, 
>>> that the integrity of this communication has been maintained nor that the 
>>> communication is free of errors, virus, interception or interference.
>>
>> 
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential, 
>> proprietary, privileged or otherwise protected by law. The message is 
>> intended solely for the named addressee. If received in error, please 
>> destroy and notify the sender. Any use of this email is prohibited when 
>> received in error. Impetus does not represent, warrant and/or guarantee, 
>> that the integrity of this communication has been maintained nor that the 
>> communication is free of errors, virus, interception or interference.
>
>
> 
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential, proprietary, 
> privileged or otherwise protected by law. The message is intended solely for 
> the named addressee. If received in error, please destroy and notify the 
> sender. Any use of this email is prohibited when received in error. Impetus 
> does not represent, warrant and/or guarantee, that the integrity of this 
> communication has been maintained nor that the communication is free of 
> errors, virus, interception or interference.
>








NOTE: This message may contain information that is confidential, proprietary, 
privileged or otherwise protected by law. The message is intended solely for 
the named addressee. If received in error, please destroy and notify the 
sender. Any use of this email is prohibited when received in error. Impetus 
does not represent, warrant and/or guarantee, that the integrity of this 
communication has been maintained nor that the communication is free of errors, 
virus, interception or interference.

[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...

[pvillard31]

Github user pvillard31 commented on the pull request:

https://github.com/apache/nifi/pull/347#issuecomment-212876061
  
@olegz Thanks for the code. However if I try to run it, it will try to 
connect to localhost:8080/nifi and will fail:

594  [NiFi Site-to-Site Connection Pool Maintenance] WARN  
org.apache.nifi.remote.client.socket.EndpointConnectionPool - 
EndpointConnectionPool[Cluster URL=http://localhost:8080/nifi] Unable to 
refresh Remote Group's peers due to {}
java.net.ConnectException: Connection refused

It makes sense since there is no NiFi instance running. Did you have in 
mind to really have an instance running or to simulate the remote end in some 
ways?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Reg: Tailing a remote file

[Simon Ball]

If you're using a rolling file name pattern you will want to tail the current 
log file, and essentially ignore the rolled history, unless you need to 
represses or miss data.

TailFile is a local processor, so you would run a nifi agent on the box that 
has the files on, use tail file -> remote process group -> input port on core 
nifi instance -> wherever your data is going next.

The local/remote setting comes up when clustering nifi and using something like 
an NFS mount to get to the files being tailed, which is a different way of 
doing this. The remote will lead to state about where you've gotten to being 
stored differently to support the cluster continuing to tail from alternative 
nifi cluster nodes.

Simon


> On 21 Apr 2016, at 11:38, Sourav Gulati  wrote:
> 
> Hi Simon/Team.
> 
> Please provide some good example of tail processor.
> 
> I need some example for " Rolling Filename Pattern".
> 
> Also, as per my knowledge TailFile processor is used for tailing local file 
> then what does location attribute signifies, if I set it to remote.
> 
> Regards,
> Sourav Gulati
> 
> -Original Message-
> From: Simon Ball [mailto:sb...@hortonworks.com]
> Sent: Wednesday, April 20, 2016 2:27 PM
> To: dev@nifi.apache.org
> Subject: Re: Reg: Tailing a remote file
> 
> Site-to-site is in pretty much every version of nifi released under apache.
> 
>> On 20 Apr 2016, at 10:48, Sourav Gulati  wrote:
>> 
>> Thanks Simon. Which version of Nifi support site to site?
>> 
>> Regards,
>> Sourav Gulati
>> 
>> -Original Message-
>> From: Simon Ball [mailto:sb...@hortonworks.com]
>> Sent: Wednesday, April 20, 2016 1:28 PM
>> To: dev@nifi.apache.org
>> Subject: Re: Reg: Tailing a remote file
>> 
>> The best way to do this would be to run a nifi agent on the remote system, 
>> using the TailFile processor, and then use site-to-site to get that into 
>> your core nifi.
>> 
>> Simon
>> 
>> 
>>> On 20 Apr 2016, at 09:49, Sourav Gulati  wrote:
>>> 
>>> Hi Team,
>>> 
>>> We want to process logs from a file which is getting updated continuously 
>>> on remote system.
>>> 
>>> What is the best way to do that in NiFi?
>>> 
>>> Regards,
>>> Sourav Gulati
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> NOTE: This message may contain information that is confidential, 
>>> proprietary, privileged or otherwise protected by law. The message is 
>>> intended solely for the named addressee. If received in error, please 
>>> destroy and notify the sender. Any use of this email is prohibited when 
>>> received in error. Impetus does not represent, warrant and/or guarantee, 
>>> that the integrity of this communication has been maintained nor that the 
>>> communication is free of errors, virus, interception or interference.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> NOTE: This message may contain information that is confidential, 
>> proprietary, privileged or otherwise protected by law. The message is 
>> intended solely for the named addressee. If received in error, please 
>> destroy and notify the sender. Any use of this email is prohibited when 
>> received in error. Impetus does not represent, warrant and/or guarantee, 
>> that the integrity of this communication has been maintained nor that the 
>> communication is free of errors, virus, interception or interference.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> NOTE: This message may contain information that is confidential, proprietary, 
> privileged or otherwise protected by law. The message is intended solely for 
> the named addressee. If received in error, please destroy and notify the 
> sender. Any use of this email is prohibited when received in error. Impetus 
> does not represent, warrant and/or guarantee, that the integrity of this 
> communication has been maintained nor that the communication is free of 
> errors, virus, interception or interference.
> 

RE: Reg: Tailing a remote file

[Sourav Gulati]

Hi Simon/Team.

Please provide some good example of tail processor.

I need some example for " Rolling Filename Pattern".

Also, as per my knowledge TailFile processor is used for tailing local file 
then what does location attribute signifies, if I set it to remote.

Regards,
Sourav Gulati

-Original Message-
From: Simon Ball [mailto:sb...@hortonworks.com]
Sent: Wednesday, April 20, 2016 2:27 PM
To: dev@nifi.apache.org
Subject: Re: Reg: Tailing a remote file

Site-to-site is in pretty much every version of nifi released under apache.

> On 20 Apr 2016, at 10:48, Sourav Gulati  wrote:
>
> Thanks Simon. Which version of Nifi support site to site?
>
> Regards,
> Sourav Gulati
>
> -Original Message-
> From: Simon Ball [mailto:sb...@hortonworks.com]
> Sent: Wednesday, April 20, 2016 1:28 PM
> To: dev@nifi.apache.org
> Subject: Re: Reg: Tailing a remote file
>
> The best way to do this would be to run a nifi agent on the remote system, 
> using the TailFile processor, and then use site-to-site to get that into your 
> core nifi.
>
> Simon
>
>
>> On 20 Apr 2016, at 09:49, Sourav Gulati  wrote:
>>
>> Hi Team,
>>
>> We want to process logs from a file which is getting updated continuously on 
>> remote system.
>>
>> What is the best way to do that in NiFi?
>>
>> Regards,
>> Sourav Gulati
>>
>>
>> 
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential, 
>> proprietary, privileged or otherwise protected by law. The message is 
>> intended solely for the named addressee. If received in error, please 
>> destroy and notify the sender. Any use of this email is prohibited when 
>> received in error. Impetus does not represent, warrant and/or guarantee, 
>> that the integrity of this communication has been maintained nor that the 
>> communication is free of errors, virus, interception or interference.
>
> 
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential, proprietary, 
> privileged or otherwise protected by law. The message is intended solely for 
> the named addressee. If received in error, please destroy and notify the 
> sender. Any use of this email is prohibited when received in error. Impetus 
> does not represent, warrant and/or guarantee, that the integrity of this 
> communication has been maintained nor that the communication is free of 
> errors, virus, interception or interference.
>









NOTE: This message may contain information that is confidential, proprietary, 
privileged or otherwise protected by law. The message is intended solely for 
the named addressee. If received in error, please destroy and notify the 
sender. Any use of this email is prohibited when received in error. Impetus 
does not represent, warrant and/or guarantee, that the integrity of this 
communication has been maintained nor that the communication is free of errors, 
virus, interception or interference.