date:20160726

[jira] [Commented] (NIFI-2208) Support Custom Properties in Expression Language - 1.x baseline

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395027#comment-15395027
 ] 

ASF GitHub Bot commented on NIFI-2208:
--

Github user YolandaMDavis commented on a diff in the pull request:

https://github.com/apache/nifi/pull/571#discussion_r72379758
  
--- Diff: nifi-mock/src/main/java/org/apache/nifi/util/TestRunners.java ---
@@ -17,11 +17,12 @@
 package org.apache.nifi.util;
 
 import org.apache.nifi.processor.Processor;
+import org.apache.nifi.registry.VariableRegistryUtils;
 
 public class TestRunners {
 
 public static TestRunner newTestRunner(final Processor processor) {
--- End diff --

agreed I went ahead and added another constructor


> Support Custom Properties in Expression Language - 1.x baseline
> ---
>
> Key: NIFI-2208
> URL: https://issues.apache.org/jira/browse/NIFI-2208
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Yolanda M. Davis
>Priority: Blocker
> Fix For: 1.0.0
>
>
> NIFI-1974 addressed this for the 0.x baseline but the PR does not apply 
> cleanly to the 1.x baseline. Creating a separate JIRA for 1.x so that we can 
> close out NIFI-1974 since 0.7.0 is ready to be released.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #571: NIFI-2208 - Support Custom Properties in Expression ...

2016-07-26 Thread YolandaMDavis

Github user YolandaMDavis commented on a diff in the pull request:

https://github.com/apache/nifi/pull/571#discussion_r72379758
  
--- Diff: nifi-mock/src/main/java/org/apache/nifi/util/TestRunners.java ---
@@ -17,11 +17,12 @@
 package org.apache.nifi.util;
 
 import org.apache.nifi.processor.Processor;
+import org.apache.nifi.registry.VariableRegistryUtils;
 
 public class TestRunners {
 
 public static TestRunner newTestRunner(final Processor processor) {
--- End diff --

agreed I went ahead and added another constructor


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2208) Support Custom Properties in Expression Language - 1.x baseline

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395026#comment-15395026
 ] 

ASF GitHub Bot commented on NIFI-2208:
--

Github user YolandaMDavis commented on the issue:

https://github.com/apache/nifi/pull/571
  

@JPercivall thanks for the review!  Just wanted to add a couple of notes on 
questions you had above:


> @YolandaMDavis, what happens if an incoming FlowFile, 
FileVariableRegistry and custom EL attributes all have the same key? Is there 
any ordering enforced? From looking at the code, I believe it to be random.
>

The order is dictated through the MultiMap's (now ImmutableMultiMap) use of 
an ArrayList internally which maintains the order in which a map was added to 
that list.  In the case you mentioned where the same key name appears across a 
flow file, file/property registry etc the first map encountered in the 
ImmutableMultiMap's list with that value will be used.

When the variable registry is initialized via VariableRegistryUtils (as it 
is in most of the application) environment variables and java application 
properties are added first, with those values taking precedence.  

The details of this will be added to the admin guide (which I believe is 
part of another issue on Jira list, but I can confirm).



> Support Custom Properties in Expression Language - 1.x baseline
> ---
>
> Key: NIFI-2208
> URL: https://issues.apache.org/jira/browse/NIFI-2208
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Yolanda M. Davis
>Priority: Blocker
> Fix For: 1.0.0
>
>
> NIFI-1974 addressed this for the 0.x baseline but the PR does not apply 
> cleanly to the 1.x baseline. Creating a separate JIRA for 1.x so that we can 
> close out NIFI-1974 since 0.7.0 is ready to be released.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #571: NIFI-2208 - Support Custom Properties in Expression Languag...

2016-07-26 Thread YolandaMDavis

Github user YolandaMDavis commented on the issue:

https://github.com/apache/nifi/pull/571
  

@JPercivall thanks for the review!  Just wanted to add a couple of notes on 
questions you had above:


> @YolandaMDavis, what happens if an incoming FlowFile, 
FileVariableRegistry and custom EL attributes all have the same key? Is there 
any ordering enforced? From looking at the code, I believe it to be random.
>

The order is dictated through the MultiMap's (now ImmutableMultiMap) use of 
an ArrayList internally which maintains the order in which a map was added to 
that list.  In the case you mentioned where the same key name appears across a 
flow file, file/property registry etc the first map encountered in the 
ImmutableMultiMap's list with that value will be used.

When the variable registry is initialized via VariableRegistryUtils (as it 
is in most of the application) environment variables and java application 
properties are added first, with those values taking precedence.  

The details of this will be added to the admin guide (which I believe is 
part of another issue on Jira list, but I can confirm).



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #571: NIFI-2208 - Support Custom Properties in Expression ...

2016-07-26 Thread YolandaMDavis

Github user YolandaMDavis commented on a diff in the pull request:

https://github.com/apache/nifi/pull/571#discussion_r72378562
  
--- Diff: 
nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/NotificationServiceManager.java
 ---
@@ -247,7 +247,7 @@ public PropertyValue getProperty(final 
PropertyDescriptor descriptor) {
 configuredValue = fullPropDescriptor.getDefaultValue();
 }
 
-return new StandardPropertyValue(configuredValue, null);
+return new StandardPropertyValue(configuredValue, null, 
null);
--- End diff --

Didn't appear to be a necessity in previous version however doesn't hurt to 
add.  Will get it done!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #571: NIFI-2208 - Support Custom Properties in Expression ...

2016-07-26 Thread YolandaMDavis

Github user YolandaMDavis commented on a diff in the pull request:

https://github.com/apache/nifi/pull/571#discussion_r72378459
  
--- Diff: 
nifi-commons/nifi-expression-language/src/main/java/org/apache/nifi/attribute/expression/language/StandardPropertyValue.java
 ---
@@ -134,9 +139,14 @@ public PropertyValue 
evaluateAttributeExpressions(final FlowFile flowFile, final
 return this;
 }
 
-return new 
StandardPropertyValue(preparedQuery.evaluateExpressions(flowFile, 
additionalAttributes, decorator), serviceLookup, null);
+VariableRegistry sysRegistry = variableRegistry != null ? 
variableRegistry : VariableRegistryUtils.createVariableRegistry();
--- End diff --

This line will change when refactor added to support 
NotificationServiceManager


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1663) Add support for ORC format

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394999#comment-15394999
 ] 

ASF GitHub Bot commented on NIFI-1663:
--

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/727

NIFI-1663: Add ConvertAvroToORC processor

This PR is based on #706 which removed the ConvertAvroToORC processor using 
Hive 2.x and Apache ORC 1.x. This PR replaces that processor with one that uses 
Hive 1.2.1 (which includes hive-orc before it was split into its own Apache 
project).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi old_orc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/727.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #727


commit fa04188470e96d979819b7bf71723cad25c25105
Author: Matt Burgess 
Date:   2016-07-21T15:59:41Z

NIFI-1868: Add PutHiveStreaming processor

commit 108fb52113e57c424516fbfdcfc6b15edc34f12b
Author: Matt Burgess 
Date:   2016-07-22T15:14:16Z

NIFI-1868: Downgrade to Hive 1.2.1 and remove ConvertAvroToORC

commit 91de9edcbc85f90f104c24f1f18caf4f4f0283d9
Author: Matt Burgess 
Date:   2016-07-27T03:25:11Z

NIFI-1663: Add ConvertAvroToORC processor




> Add support for ORC format
> --
>
> Key: NIFI-1663
> URL: https://issues.apache.org/jira/browse/NIFI-1663
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.0.0
>
>
> From the Hive/ORC wiki 
> (https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC): 
> The Optimized Row Columnar (ORC) file format provides a highly efficient way 
> to store Hive data ... Using ORC files improves performance when Hive is 
> reading, writing, and processing data.
> As users are interested in NiFi integrations with Hive (NIFI-981, NIFI-1193, 
> etc.), NiFi should be able to support ORC file format to enable users to 
> efficiently store flow files for use by Hive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #727: NIFI-1663: Add ConvertAvroToORC processor

2016-07-26 Thread mattyb149

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/727

NIFI-1663: Add ConvertAvroToORC processor

This PR is based on #706 which removed the ConvertAvroToORC processor using 
Hive 2.x and Apache ORC 1.x. This PR replaces that processor with one that uses 
Hive 1.2.1 (which includes hive-orc before it was split into its own Apache 
project).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi old_orc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/727.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #727


commit fa04188470e96d979819b7bf71723cad25c25105
Author: Matt Burgess 
Date:   2016-07-21T15:59:41Z

NIFI-1868: Add PutHiveStreaming processor

commit 108fb52113e57c424516fbfdcfc6b15edc34f12b
Author: Matt Burgess 
Date:   2016-07-22T15:14:16Z

NIFI-1868: Downgrade to Hive 1.2.1 and remove ConvertAvroToORC

commit 91de9edcbc85f90f104c24f1f18caf4f4f0283d9
Author: Matt Burgess 
Date:   2016-07-27T03:25:11Z

NIFI-1663: Add ConvertAvroToORC processor




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Issue Comment Deleted] (NIFI-2410) PG level controller service display issue

2016-07-26 Thread Haimo Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haimo Liu updated NIFI-2410:

Comment: was deleted

(was: controller settings -> CS)

> PG level controller service display issue
> -
>
> Key: NIFI-2410
> URL: https://issues.apache.org/jira/browse/NIFI-2410
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Priority: Critical
> Fix For: 1.0.0
>
> Attachments: Screen Shot 2016-07-26 at 10.36.08 PM.png, Screen Shot 
> 2016-07-26 at 10.57.19 PM.png, Screen Shot 2016-07-26 at 11.05.42 PM.png
>
>
> NIFI 1.0 snapshot, 7/26 build, didn't turn on internal authorization/policy 
> management, running in a clustered mode (3-node).
> Group level CS display seems inconsistent
> 1. menu -> Controller Settings -> Controller Services, can see one CS (no 
> matter which PG i am in, can always see this single CS, "DBCPConnectionPool", 
> see attached image)
> 2. Drill in to a PG where I have the GenerateTableFetch processor, can see 
> five different controller services here, presumably all PG level CS
> 3. create a new PG, drag & drop GenerateTableFetch processor, can see two 
> controller services "MySqlConnectionPool" and another "MySqlConnectionPool"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2410) PG level controller service display issue

2016-07-26 Thread Haimo Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haimo Liu updated NIFI-2410:

Attachment: Screen Shot 2016-07-26 at 10.57.19 PM.png

> PG level controller service display issue
> -
>
> Key: NIFI-2410
> URL: https://issues.apache.org/jira/browse/NIFI-2410
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Priority: Critical
> Fix For: 1.0.0
>
> Attachments: Screen Shot 2016-07-26 at 10.36.08 PM.png, Screen Shot 
> 2016-07-26 at 10.57.19 PM.png, Screen Shot 2016-07-26 at 11.05.42 PM.png
>
>
> NIFI 1.0 snapshot, 7/26 build, didn't turn on internal authorization/policy 
> management, running in a clustered mode (3-node).
> Group level CS display seems inconsistent
> 1. menu -> Controller Settings -> Controller Services, can see one CS (no 
> matter which PG i am in, can always see this single CS, "DBCPConnectionPool", 
> see attached image)
> 2. Drill in to a PG where I have the GenerateTableFetch processor, can see 
> five different controller services here, presumably all PG level CS
> 3. create a new PG, drag & drop GenerateTableFetch processor, can see two 
> controller services "MySqlConnectionPool" and another "MySqlConnectionPool"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2410) PG level controller service display issue

2016-07-26 Thread Haimo Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haimo Liu updated NIFI-2410:

Attachment: Screen Shot 2016-07-26 at 11.05.42 PM.png

> PG level controller service display issue
> -
>
> Key: NIFI-2410
> URL: https://issues.apache.org/jira/browse/NIFI-2410
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Priority: Critical
> Fix For: 1.0.0
>
> Attachments: Screen Shot 2016-07-26 at 10.36.08 PM.png, Screen Shot 
> 2016-07-26 at 10.57.19 PM.png, Screen Shot 2016-07-26 at 11.05.42 PM.png
>
>
> NIFI 1.0 snapshot, 7/26 build, didn't turn on internal authorization/policy 
> management, running in a clustered mode (3-node).
> Group level CS display seems inconsistent
> 1. menu -> Controller Settings -> Controller Services, can see one CS (no 
> matter which PG i am in, can always see this single CS, "DBCPConnectionPool", 
> see attached image)
> 2. Drill in to a PG where I have the GenerateTableFetch processor, can see 
> five different controller services here, presumably all PG level CS
> 3. create a new PG, drag & drop GenerateTableFetch processor, can see two 
> controller services "MySqlConnectionPool" and another "MySqlConnectionPool"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2410) PG level controller service display issue

2016-07-26 Thread Haimo Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haimo Liu updated NIFI-2410:

Attachment: Screen Shot 2016-07-26 at 10.36.08 PM.png

controller settings -> CS

> PG level controller service display issue
> -
>
> Key: NIFI-2410
> URL: https://issues.apache.org/jira/browse/NIFI-2410
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Priority: Critical
> Fix For: 1.0.0
>
> Attachments: Screen Shot 2016-07-26 at 10.36.08 PM.png
>
>
> NIFI 1.0 snapshot, 7/26 build, didn't turn on internal authorization/policy 
> management, running in a clustered mode (3-node).
> Group level CS display seems inconsistent
> 1. menu -> Controller Settings -> Controller Services, can see one CS (no 
> matter which PG i am in, can always see this single CS, "DBCPConnectionPool", 
> see attached image)
> 2. Drill in to a PG where I have the GenerateTableFetch processor, can see 
> five different controller services here, presumably all PG level CS
> 3. create a new PG, drag & drop GenerateTableFetch processor, can see two 
> controller services "MySqlConnectionPool" and another "MySqlConnectionPool"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2410) PG level controller service display issue

2016-07-26 Thread Haimo Liu (JIRA)

Haimo Liu created NIFI-2410:
---

 Summary: PG level controller service display issue
 Key: NIFI-2410
 URL: https://issues.apache.org/jira/browse/NIFI-2410
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.0.0
Reporter: Haimo Liu
Priority: Critical
 Fix For: 1.0.0


NIFI 1.0 snapshot, 7/26 build, didn't turn on internal authorization/policy 
management, running in a clustered mode (3-node).

Group level CS display seems inconsistent

1. menu -> Controller Settings -> Controller Services, can see one CS (no 
matter which PG i am in, can always see this single CS, "DBCPConnectionPool", 
see attached image)

2. Drill in to a PG where I have the GenerateTableFetch processor, can see five 
different controller services here, presumably all PG level CS

3. create a new PG, drag & drop GenerateTableFetch processor, can see two 
controller services "MySqlConnectionPool" and another "MySqlConnectionPool"







--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1465) Upgrade encryption of sensitive properties

2016-07-26 Thread Andy LoPresto (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-1465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394965#comment-15394965
 ] 

Andy LoPresto commented on NIFI-1465:
-

A lot of additional context from this email thread [1]. 

{quote}
Brett,

(I added the dev list back in because this is probably of interest to someone/ 
should be documented. )

This started out as a brief email and spiraled into a couple days of work. Part 
of that is because my Ruby is rusty, but part is because there are some serious 
underlying issues here (previously quickly noted in NIFI-1465 [1] but not fully 
expounded upon at the time). The tl;dr of this email is that I have written a 
Ruby script which will accomplish what you want and it is located here [2]. 
Read on at your own risk (various parts of this email were written over the 
course of the last two days, so it may be repetitive/incoherent where the story 
changed). 

This is a confusing case because it is unusual in execution and there are 
multiple layers here. I think I didn’t explain it well last time. I’ll try to 
step through it, but I also apologize in advance, because I found a lot of 
legacy stuff here that was probably written with no intention of ever being 
exposed/integrated with an external source. Kerckhoff would not approve. I will 
add some of this to NIFI-1465. 

The mechanism that NiFi uses to encrypt the sensitive property values (i.e. 
passwords for EncryptContent) is as I described previously, but in further 
investigating to help solve your problem, I realized that the Jasypt [3] 
StandardPBEStringEncryptor used in StandardFlowSynchronizer uses a random salt 
generator internally. You can verify this by making a new flow with two 
EncryptContent processors — even if you set the same password for each, the 
resulting cipher texts in the flow.xml will be unique because despite the same 
master key being used, the random salt will cause them to be different. 

Now this is actually good news, because it means the salt must be encoded and 
transmitted with the cipher text. If it was not, NiFi would not be able to 
decrypt these values unless it used a fixed salt, and clearly it does not. So 
as long as your Ruby code generates a salt of the correct length and embeds it 
in the cipher text, it will be compatible with NiFi. The salt is the first 16 
bytes (32 hex characters) and the actual cipher text representing the encrypted 
processor sensitive property (happens to be another password, but this is 
irrelevant) is the second 16 bytes. 

Note: because your initial plaintext (the password you are trying to encrypt) 
is only 11 UTF-8 characters, it can be represented by 11 bytes. This means that 
when encrypted using AES-CBC (16 byte block size), it requires only and exactly 
one block (11 bytes of plaintext plus 5 bytes padding). The resulting cipher 
text is 16 bytes. If the processor password was longer than 16 characters, it 
would be encrypted in two blocks and encoded as 32 bytes, or 64 hex characters 
alone (remember to add the initial 32 chars for the salt for a total of 96 
chars). 

What we are looking for as the output of the Ruby operation is, as you noted, a 
64 character hex string, of the format:

output = hex_encode(salt || encrypt(processor_password, master_key, iv))

where the master_key and iv are derived by

(master_key, iv) = md5(master_passphrase || master_salt) || 
md5(md5(master_passphrase || master_salt) || master_passphrase || master_salt) 
|| md5(md5(md5(…)…)…)

This is an unusual method and is described thusly on the OpenSSL EVP_BytesToKey 
documentation [4]: 

If the total key and IV length is less than the digest length and MD5 is used 
then the derivation algorithm is compatible with PKCS#5 v1.5 otherwise a non 
standard extension is used to derive the extra data.

...

KEY DERIVATION ALGORITHM

The key and IV is derived by concatenating D_1, D_2, etc until enough data is 
available for the key and IV. D_i is defined as:

D_i = HASH^count(D_(i-1) || data || salt)
where || denotes concatenation, D_0 is empty, HASH is the digest algorithm in 
use, HASH^1(data) is simply HASH(data), HASH^2(data) is HASH(HASH(data)) and so 
on.

The initial bytes are used for the key and the subsequent bytes for the IV.

The reason there are multiple MD5 operations above is because we have specified 
the encryption will use AES-256-CBC, which requires a 256 bit (32 byte) key — 
32 bytes are represented by 64 hex characters. A single iteration of MD5 only 
yields 16 bytes (32 hex chars), so we must concatenate it with another 
invocation. However, as it is deterministic, running it on the same input would 
return the same output, and the key would just repeat the same 16 bytes. To 
counter this, the second ( up to n many) invocation “salts” the input with the 
result of the previous step. If we substitute some variables for the full 
expression above, we can see this more

[jira] [Commented] (NIFI-2409) the dataset url says is invalid, even though it is a valid dataset url

2016-07-26 Thread pradeep arumalla (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394961#comment-15394961
 ] 

pradeep arumalla commented on NIFI-2409:


will add screen shots shortly.

> the dataset url says is invalid, even though it is a valid dataset url
> --
>
> Key: NIFI-2409
> URL: https://issues.apache.org/jira/browse/NIFI-2409
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 0.6.1
>Reporter: pradeep arumalla
>Priority: Blocker
>
> hi there, I am using StoreInKiteDataset , and when I give the  dataset URI as 
> "dataset:hdfs:/user/xxx/dataset" path, it says "is invalid because dataset 
> URI is invalid:unknown dataset URI", not sure why its saying invalid.The same 
> dataset url worked a week ago.
> I used Kite command 
> >./kite-dataset create dataset:hdfs:/user/xxx/dataset --schema  
> >hdfs:/user/xxx/schemas/yy.avsc  --format parquet 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

2016-07-26 Thread Joseph Witt (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394956#comment-15394956
 ] 

Joseph Witt commented on NIFI-2395:
---

To be stuck here 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1370

This means that there continue to be more journalFiles than the allowed 
threshold.  This log statement should be showing up in the nifi-app.log over 
and over then.
  
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1342

That would be good to see the output of at least in terms of the variables it 
would provide (journal files vs journal file threshold) over time.

Also, please verify whether you see any of the following ERROR entries in your 
log file ""Failed to merge Journal Files ".  If yes there should be some very 
useful information there followed by a stack trace.  Do you see this?



> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Blocker
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} bytes)", 
> journalFileCount, repoSize, journalCountThreshold, sizeThreshold);
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> }
> logger.info("Provenance Repository has now caught up with 
> rolling over journal files. Current number of "
> + "journal files to be rolled over is {}", 
> journalFileCount);
> }
> {code}
> My nifi is at the sleep indefinitely.  The reason my nifi cannot move forward 
> is because of the thread doing the merge is stopped.  The thread doing the 
> merge is at:
> {code}
> accepted = eventQueue.offer(new Tuple<>(record, blockIndex), 10, 
> TimeUnit.MILLISECONDS);
> {code}
> so the queue is full.  
> What I believe happened is that the callables created here:
> {code}
> final Callable callable = new 
> Callable() {
> @Override
> public Object call() throws IOException {
> while

[jira] [Comment Edited] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

2016-07-26 Thread Joseph Witt (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394933#comment-15394933
 ] 

Joseph Witt edited comment on NIFI-2395 at 7/27/16 2:07 AM:


also [~badavis] can you please share the configuration settings you have in 
nifi.properties for the following 

{quote}
nifi.provenance.repository.directory.prov1=/repos/prov/prov-repo1
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=50 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=6
nifi.provenance.repository.indexing.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
\# Comma-separated list of fields. Fields that are not indexed will not be 
searchable. Valid fields are:
\# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, 
AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, 
ProcessorID
\# FlowFile Attributes that should be indexed and made searchable
nifi.provenance.repository.indexed.attributes=twitter.msg, language
\# Large values for the shard size will result in more Java heap usage when 
searching the Provenance Repository
\# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
{quote}


was (Author: joewitt):
also [~badavis] can you please share the configuration settings you have in 
nifi.properties for the following 

{quote}
nifi.provenance.repository.directory.prov1=/repos/prov/prov-repo1
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=50 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=6
nifi.provenance.repository.indexing.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
# Comma-separated list of fields. Fields that are not indexed will not be 
searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, 
AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, 
ProcessorID
# FlowFile Attributes that should be indexed and made searchable
nifi.provenance.repository.indexed.attributes=twitter.msg, language
# Large values for the shard size will result in more Java heap usage when 
searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
{quote}

> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Blocker
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due

[jira] [Commented] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

2016-07-26 Thread Joseph Witt (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394933#comment-15394933
 ] 

Joseph Witt commented on NIFI-2395:
---

also [~badavis] can you please share the configuration settings you have in 
nifi.properties for the following 

{quote}
nifi.provenance.repository.directory.prov1=/repos/prov/prov-repo1
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=50 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=6
nifi.provenance.repository.indexing.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
# Comma-separated list of fields. Fields that are not indexed will not be 
searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, 
AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, 
ProcessorID
# FlowFile Attributes that should be indexed and made searchable
nifi.provenance.repository.indexed.attributes=twitter.msg, language
# Large values for the shard size will result in more Java heap usage when 
searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
{quote}

> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Blocker
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} bytes)", 
> journalFileCount, repoSize, journalCountThreshold, sizeThreshold);
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> }
> logger.info("Provenance Repository has now caught up with 
> rolling over journal files. Current number of "
> + "journal files to be rolled over is {}", 
> journalFileCount);
> }
> {code}
> My nifi is at the sleep indefinitely.  The reason my nifi cannot move forward 
> is because of the thread doing the merge is stopped.  The thread doing the 
> merge is at:
> {code}
> accepted = eventQueue.offer(new Tuple<>(record, blockIndex), 10, 
> TimeUnit.MILLISECONDS);
> {code}
> so the queue is full.  
> What I believe happened is that the callables

[jira] [Commented] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

2016-07-26 Thread Joseph Witt (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394929#comment-15394929
 ] 

Joseph Witt commented on NIFI-2395:
---

[~badavis] can you please confirm that you're seeing these WARNs in your logs 
(and really in the UI too) with "The rate of the dataflow is exceeding the 
provenance recording rate."

> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Blocker
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} bytes)", 
> journalFileCount, repoSize, journalCountThreshold, sizeThreshold);
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> }
> logger.info("Provenance Repository has now caught up with 
> rolling over journal files. Current number of "
> + "journal files to be rolled over is {}", 
> journalFileCount);
> }
> {code}
> My nifi is at the sleep indefinitely.  The reason my nifi cannot move forward 
> is because of the thread doing the merge is stopped.  The thread doing the 
> merge is at:
> {code}
> accepted = eventQueue.offer(new Tuple<>(record, blockIndex), 10, 
> TimeUnit.MILLISECONDS);
> {code}
> so the queue is full.  
> What I believe happened is that the callables created here:
> {code}
> final Callable callable = new 
> Callable() {
> @Override
> public Object call() throws IOException {
> while (!eventQueue.isEmpty() || 
> !finishedAdding.get()) {
> final 
> Tuple tuple;
> try {
> tuple = eventQueue.poll(10, 
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException 
> ie) {
> continue;
> }
> if (tuple == null) {
> continue;
> }
> indexingAction.index(tuple.getKey(), 
> indexWriter, tuple.getValue());
> }
>

[jira] [Commented] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

2016-07-26 Thread Joseph Witt (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394927#comment-15394927
 ] 

Joseph Witt commented on NIFI-2395:
---

raising to blocker until it can be proven invalid somehow or the issue can be 
identified.  The analysis is quite reasonable so there is clearly something at 
play.  If we can get a thread dump that would be super helpful.

> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Blocker
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} bytes)", 
> journalFileCount, repoSize, journalCountThreshold, sizeThreshold);
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> }
> logger.info("Provenance Repository has now caught up with 
> rolling over journal files. Current number of "
> + "journal files to be rolled over is {}", 
> journalFileCount);
> }
> {code}
> My nifi is at the sleep indefinitely.  The reason my nifi cannot move forward 
> is because of the thread doing the merge is stopped.  The thread doing the 
> merge is at:
> {code}
> accepted = eventQueue.offer(new Tuple<>(record, blockIndex), 10, 
> TimeUnit.MILLISECONDS);
> {code}
> so the queue is full.  
> What I believe happened is that the callables created here:
> {code}
> final Callable callable = new 
> Callable() {
> @Override
> public Object call() throws IOException {
> while (!eventQueue.isEmpty() || 
> !finishedAdding.get()) {
> final 
> Tuple tuple;
> try {
> tuple = eventQueue.poll(10, 
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException 
> ie) {
> continue;
> }
> if (tuple == null) {
> continue;
> }
> indexingAction.index(tuple.getKey(), 
> indexWriter, tuple.getValue());
>

[jira] [Updated] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

2016-07-26 Thread Joseph Witt (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2395:
--
Affects Version/s: 0.7.0

> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Critical
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} bytes)", 
> journalFileCount, repoSize, journalCountThreshold, sizeThreshold);
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> }
> logger.info("Provenance Repository has now caught up with 
> rolling over journal files. Current number of "
> + "journal files to be rolled over is {}", 
> journalFileCount);
> }
> {code}
> My nifi is at the sleep indefinitely.  The reason my nifi cannot move forward 
> is because of the thread doing the merge is stopped.  The thread doing the 
> merge is at:
> {code}
> accepted = eventQueue.offer(new Tuple<>(record, blockIndex), 10, 
> TimeUnit.MILLISECONDS);
> {code}
> so the queue is full.  
> What I believe happened is that the callables created here:
> {code}
> final Callable callable = new 
> Callable() {
> @Override
> public Object call() throws IOException {
> while (!eventQueue.isEmpty() || 
> !finishedAdding.get()) {
> final 
> Tuple tuple;
> try {
> tuple = eventQueue.poll(10, 
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException 
> ie) {
> continue;
> }
> if (tuple == null) {
> continue;
> }
> indexingAction.index(tuple.getKey(), 
> indexWriter, tuple.getValue());
> }
> return null;
> }
> {code}
> finish before the offer adds its first event because I do not see any Index 
> Provenance Events threads.

[jira] [Updated] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

2016-07-26 Thread Joseph Witt (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2395:
--
Priority: Blocker  (was: Critical)

> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Blocker
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} bytes)", 
> journalFileCount, repoSize, journalCountThreshold, sizeThreshold);
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> }
> logger.info("Provenance Repository has now caught up with 
> rolling over journal files. Current number of "
> + "journal files to be rolled over is {}", 
> journalFileCount);
> }
> {code}
> My nifi is at the sleep indefinitely.  The reason my nifi cannot move forward 
> is because of the thread doing the merge is stopped.  The thread doing the 
> merge is at:
> {code}
> accepted = eventQueue.offer(new Tuple<>(record, blockIndex), 10, 
> TimeUnit.MILLISECONDS);
> {code}
> so the queue is full.  
> What I believe happened is that the callables created here:
> {code}
> final Callable callable = new 
> Callable() {
> @Override
> public Object call() throws IOException {
> while (!eventQueue.isEmpty() || 
> !finishedAdding.get()) {
> final 
> Tuple tuple;
> try {
> tuple = eventQueue.poll(10, 
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException 
> ie) {
> continue;
> }
> if (tuple == null) {
> continue;
> }
> indexingAction.index(tuple.getKey(), 
> indexWriter, tuple.getValue());
> }
> return null;
> }
> {code}
> finish before the offer adds its first event because I do not see any Index 
> Provenance Events

[jira] [Assigned] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

2016-07-26 Thread Joseph Witt (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt reassigned NIFI-2395:
-

Assignee: Joseph Witt

> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Critical
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} bytes)", 
> journalFileCount, repoSize, journalCountThreshold, sizeThreshold);
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> }
> logger.info("Provenance Repository has now caught up with 
> rolling over journal files. Current number of "
> + "journal files to be rolled over is {}", 
> journalFileCount);
> }
> {code}
> My nifi is at the sleep indefinitely.  The reason my nifi cannot move forward 
> is because of the thread doing the merge is stopped.  The thread doing the 
> merge is at:
> {code}
> accepted = eventQueue.offer(new Tuple<>(record, blockIndex), 10, 
> TimeUnit.MILLISECONDS);
> {code}
> so the queue is full.  
> What I believe happened is that the callables created here:
> {code}
> final Callable callable = new 
> Callable() {
> @Override
> public Object call() throws IOException {
> while (!eventQueue.isEmpty() || 
> !finishedAdding.get()) {
> final 
> Tuple tuple;
> try {
> tuple = eventQueue.poll(10, 
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException 
> ie) {
> continue;
> }
> if (tuple == null) {
> continue;
> }
> indexingAction.index(tuple.getKey(), 
> indexWriter, tuple.getValue());
> }
> return null;
> }
> {code}
> finish before the offer adds its first event because I do not see any Index 
> Provenance Events threads.  My

[GitHub] nifi pull request #655: DataDog support added

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/655#discussion_r72347152
  
--- Diff: nifi-assembly/LICENSE ---
@@ -1487,3 +1487,56 @@ This project bundles 'Jython' which is available 
under a Python Software Foundat
 OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+This project bundles âMockitoâ which is available under an MIT license
--- End diff --

This should not be added to the Assembly License. Mockito is a test dep 
only and thus should not be included in the Licensing of the binary 
distribution.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #655: DataDog support added

2016-07-26 Thread irabinovitch

Github user irabinovitch commented on a diff in the pull request:

https://github.com/apache/nifi/pull/655#discussion_r72345879
  
--- Diff: 
nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java/org/apache/nifi/reporting/datadog/DataDogReportingTask.java
 ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.reporting.datadog;
+
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.MetricRegistry;
+import com.google.common.base.Optional;
+import com.google.common.util.concurrent.AtomicDouble;
+import com.yammer.metrics.core.VirtualMachineMetrics;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.status.ProcessGroupStatus;
+import org.apache.nifi.controller.status.ProcessorStatus;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.AbstractReportingTask;
+import org.apache.nifi.reporting.ReportingContext;
+import org.apache.nifi.reporting.datadog.metrics.MetricsService;
+import org.coursera.metrics.datadog.DynamicTagsCallback;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+@Tags({"reporting", "datadog", "metrics"})
+@CapabilityDescription("Publishes metrics from NiFi to datadog")
+public class DataDogReportingTask extends AbstractReportingTask {
+
+static final AllowableValue DATADOG_AGENT = new 
AllowableValue("DataDog Agent", "DataDog UDP Agent",
+"Metrics will be sent via locally installed DataDog agent over 
UDP");
--- End diff --

UDP only used when posting the local agent. The agent in turn forward over 
HTTP to the Datadog ingest APIs.

Locally you're pushing to datadog via statsd which is a fairly standard  
metrics protocol over UDP.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2348) When processor is selected, cannot copy id from palette

2016-07-26 Thread Scott Aslan (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394617#comment-15394617
 ] 

Scott Aslan commented on NIFI-2348:
---

Joe, the use case for the copy/paste buttons on the operate palette is for 
copying and pasting processors not the UUID. As such I think we can resolve 
this as a "Wont Fix" 

> When processor is selected, cannot copy id from palette
> ---
>
> Key: NIFI-2348
> URL: https://issues.apache.org/jira/browse/NIFI-2348
> Project: Apache NiFi
>  Issue Type: Sub-task
>Reporter: Joseph Percivall
>Priority: Minor
> Fix For: 1.0.0
>
>
> Processor selected
> Processor details show up in palette
> highlight uuid in palette (processor remains "selected" on graph)
> copy then copies the processor instead of the uuid
> As a user, I would like to copy the uuid when I have the text highlighted 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #655: DataDog support added

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/655#discussion_r72345217
  
--- Diff: 
nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java/org/apache/nifi/reporting/datadog/metrics/MetricsService.java
 ---
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.reporting.datadog.metrics;
+
+import com.yammer.metrics.core.VirtualMachineMetrics;
+import org.apache.nifi.controller.status.ProcessGroupStatus;
+import org.apache.nifi.controller.status.ProcessorStatus;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A service used to produce key/value metrics based on a given input.
+ */
+public class MetricsService {
+
+//processor - specific metrics
+public Map getProcessorMetrics(ProcessorStatus status) 
{
+final Map metrics = new HashMap<>();
+metrics.put(MetricNames.FLOW_FILES_RECEIVED, 
String.valueOf(status.getInputCount()));
+metrics.put(MetricNames.FLOW_FILES_SENT, 
String.valueOf(status.getOutputCount()));
+metrics.put(MetricNames.BYTES_READ, 
String.valueOf(status.getInputBytes()));
+metrics.put(MetricNames.BYTES_WRITTEN, 
String.valueOf(status.getOutputBytes()));
+metrics.put(MetricNames.ACTIVE_THREADS, 
String.valueOf(status.getActiveThreadCount()));
--- End diff --

I'd suggest adding "status.getProcessingNanos()" as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #655: DataDog support added

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/655#discussion_r72344471
  
--- Diff: 
nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java/org/apache/nifi/reporting/datadog/DataDogReportingTask.java
 ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.reporting.datadog;
+
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.MetricRegistry;
+import com.google.common.base.Optional;
+import com.google.common.util.concurrent.AtomicDouble;
+import com.yammer.metrics.core.VirtualMachineMetrics;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.status.ProcessGroupStatus;
+import org.apache.nifi.controller.status.ProcessorStatus;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.AbstractReportingTask;
+import org.apache.nifi.reporting.ReportingContext;
+import org.apache.nifi.reporting.datadog.metrics.MetricsService;
+import org.coursera.metrics.datadog.DynamicTagsCallback;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+@Tags({"reporting", "datadog", "metrics"})
+@CapabilityDescription("Publishes metrics from NiFi to datadog")
+public class DataDogReportingTask extends AbstractReportingTask {
+
+static final AllowableValue DATADOG_AGENT = new 
AllowableValue("DataDog Agent", "DataDog UDP Agent",
+"Metrics will be sent via locally installed DataDog agent over 
UDP");
+
+static final AllowableValue DATADOG_HTTP = new AllowableValue("Datadog 
HTTP", "Datadog HTTP",
+"Metrics will be sent via HTTP transport with no need of Agent 
installed. " +
+"DataDog API key needs to be set");
+
+static final PropertyDescriptor DATADOG_TRANSPORT = new 
PropertyDescriptor.Builder()
+.name("DataDog transport")
+.description("Transport through which metrics will be sent to 
DataDog")
+.required(true)
+.allowableValues(DATADOG_AGENT, DATADOG_HTTP)
+.defaultValue(DATADOG_AGENT.getValue())
+.build();
+
+static final PropertyDescriptor API_KEY = new 
PropertyDescriptor.Builder()
+.name("API key")
+.description("DataDog API key. If specified value is 'agent', 
local DataDog agent will be used.")
+.expressionLanguageSupported(false)
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor METRICS_PREFIX = new 
PropertyDescriptor.Builder()
+.name("Metrics prefix")
+.description("Prefix to be added before every metric")
+.required(true)
+.expressionLanguageSupported(false)
+.defaultValue("nifi")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor ENVIRONMENT = new 
PropertyDescriptor.Builder()
+.name("Environment")
+.description("Environment, dataflow is running in. " +
+"This property will be included as metrics tag.")
+.required(true)
+.expressionLanguageSupported(false)
+.defaultValue("dev")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+private

[GitHub] nifi pull request #655: DataDog support added

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/655#discussion_r72343405
  
--- Diff: 
nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java/org/apache/nifi/reporting/datadog/DataDogReportingTask.java
 ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.reporting.datadog;
+
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.MetricRegistry;
+import com.google.common.base.Optional;
+import com.google.common.util.concurrent.AtomicDouble;
+import com.yammer.metrics.core.VirtualMachineMetrics;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.status.ProcessGroupStatus;
+import org.apache.nifi.controller.status.ProcessorStatus;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.AbstractReportingTask;
+import org.apache.nifi.reporting.ReportingContext;
+import org.apache.nifi.reporting.datadog.metrics.MetricsService;
+import org.coursera.metrics.datadog.DynamicTagsCallback;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+@Tags({"reporting", "datadog", "metrics"})
+@CapabilityDescription("Publishes metrics from NiFi to datadog")
+public class DataDogReportingTask extends AbstractReportingTask {
+
+static final AllowableValue DATADOG_AGENT = new 
AllowableValue("DataDog Agent", "DataDog UDP Agent",
+"Metrics will be sent via locally installed DataDog agent over 
UDP");
+
+static final AllowableValue DATADOG_HTTP = new AllowableValue("Datadog 
HTTP", "Datadog HTTP",
+"Metrics will be sent via HTTP transport with no need of Agent 
installed. " +
+"DataDog API key needs to be set");
+
+static final PropertyDescriptor DATADOG_TRANSPORT = new 
PropertyDescriptor.Builder()
+.name("DataDog transport")
+.description("Transport through which metrics will be sent to 
DataDog")
+.required(true)
+.allowableValues(DATADOG_AGENT, DATADOG_HTTP)
+.defaultValue(DATADOG_AGENT.getValue())
+.build();
+
+static final PropertyDescriptor API_KEY = new 
PropertyDescriptor.Builder()
+.name("API key")
+.description("DataDog API key. If specified value is 'agent', 
local DataDog agent will be used.")
+.expressionLanguageSupported(false)
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor METRICS_PREFIX = new 
PropertyDescriptor.Builder()
+.name("Metrics prefix")
+.description("Prefix to be added before every metric")
+.required(true)
+.expressionLanguageSupported(false)
+.defaultValue("nifi")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor ENVIRONMENT = new 
PropertyDescriptor.Builder()
+.name("Environment")
+.description("Environment, dataflow is running in. " +
+"This property will be included as metrics tag.")
+.required(true)
+.expressionLanguageSupported(false)
+.defaultValue("dev")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+private

[GitHub] nifi pull request #655: DataDog support added

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/655#discussion_r72342803
  
--- Diff: 
nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java/org/apache/nifi/reporting/datadog/DataDogReportingTask.java
 ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.reporting.datadog;
+
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.MetricRegistry;
+import com.google.common.base.Optional;
+import com.google.common.util.concurrent.AtomicDouble;
+import com.yammer.metrics.core.VirtualMachineMetrics;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.status.ProcessGroupStatus;
+import org.apache.nifi.controller.status.ProcessorStatus;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.AbstractReportingTask;
+import org.apache.nifi.reporting.ReportingContext;
+import org.apache.nifi.reporting.datadog.metrics.MetricsService;
+import org.coursera.metrics.datadog.DynamicTagsCallback;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+@Tags({"reporting", "datadog", "metrics"})
+@CapabilityDescription("Publishes metrics from NiFi to datadog")
+public class DataDogReportingTask extends AbstractReportingTask {
+
+static final AllowableValue DATADOG_AGENT = new 
AllowableValue("DataDog Agent", "DataDog UDP Agent",
+"Metrics will be sent via locally installed DataDog agent over 
UDP");
+
+static final AllowableValue DATADOG_HTTP = new AllowableValue("Datadog 
HTTP", "Datadog HTTP",
+"Metrics will be sent via HTTP transport with no need of Agent 
installed. " +
+"DataDog API key needs to be set");
+
+static final PropertyDescriptor DATADOG_TRANSPORT = new 
PropertyDescriptor.Builder()
+.name("DataDog transport")
+.description("Transport through which metrics will be sent to 
DataDog")
+.required(true)
+.allowableValues(DATADOG_AGENT, DATADOG_HTTP)
+.defaultValue(DATADOG_AGENT.getValue())
+.build();
+
+static final PropertyDescriptor API_KEY = new 
PropertyDescriptor.Builder()
+.name("API key")
+.description("DataDog API key. If specified value is 'agent', 
local DataDog agent will be used.")
+.expressionLanguageSupported(false)
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor METRICS_PREFIX = new 
PropertyDescriptor.Builder()
+.name("Metrics prefix")
+.description("Prefix to be added before every metric")
+.required(true)
+.expressionLanguageSupported(false)
+.defaultValue("nifi")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor ENVIRONMENT = new 
PropertyDescriptor.Builder()
+.name("Environment")
+.description("Environment, dataflow is running in. " +
+"This property will be included as metrics tag.")
+.required(true)
+.expressionLanguageSupported(false)
--- End diff --

Kinda contrary to a previous comment, I'd suggest making this support EL 
(and then adding back in the evaluation). This is

[GitHub] nifi pull request #655: DataDog support added

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/655#discussion_r72342292
  
--- Diff: 
nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java/org/apache/nifi/reporting/datadog/DataDogReportingTask.java
 ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.reporting.datadog;
+
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.MetricRegistry;
+import com.google.common.base.Optional;
+import com.google.common.util.concurrent.AtomicDouble;
+import com.yammer.metrics.core.VirtualMachineMetrics;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.status.ProcessGroupStatus;
+import org.apache.nifi.controller.status.ProcessorStatus;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.AbstractReportingTask;
+import org.apache.nifi.reporting.ReportingContext;
+import org.apache.nifi.reporting.datadog.metrics.MetricsService;
+import org.coursera.metrics.datadog.DynamicTagsCallback;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+@Tags({"reporting", "datadog", "metrics"})
+@CapabilityDescription("Publishes metrics from NiFi to datadog")
+public class DataDogReportingTask extends AbstractReportingTask {
+
+static final AllowableValue DATADOG_AGENT = new 
AllowableValue("DataDog Agent", "DataDog UDP Agent",
+"Metrics will be sent via locally installed DataDog agent over 
UDP");
+
+static final AllowableValue DATADOG_HTTP = new AllowableValue("Datadog 
HTTP", "Datadog HTTP",
+"Metrics will be sent via HTTP transport with no need of Agent 
installed. " +
+"DataDog API key needs to be set");
+
+static final PropertyDescriptor DATADOG_TRANSPORT = new 
PropertyDescriptor.Builder()
+.name("DataDog transport")
+.description("Transport through which metrics will be sent to 
DataDog")
+.required(true)
+.allowableValues(DATADOG_AGENT, DATADOG_HTTP)
+.defaultValue(DATADOG_AGENT.getValue())
--- End diff --

I'd prefer DATADOG_HTTP be made the default value since it does not require 
an external thing to be installed on the local system.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #655: DataDog support added

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/655#discussion_r72342175
  
--- Diff: 
nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java/org/apache/nifi/reporting/datadog/DataDogReportingTask.java
 ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.reporting.datadog;
+
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.MetricRegistry;
+import com.google.common.base.Optional;
+import com.google.common.util.concurrent.AtomicDouble;
+import com.yammer.metrics.core.VirtualMachineMetrics;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.status.ProcessGroupStatus;
+import org.apache.nifi.controller.status.ProcessorStatus;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.AbstractReportingTask;
+import org.apache.nifi.reporting.ReportingContext;
+import org.apache.nifi.reporting.datadog.metrics.MetricsService;
+import org.coursera.metrics.datadog.DynamicTagsCallback;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+@Tags({"reporting", "datadog", "metrics"})
+@CapabilityDescription("Publishes metrics from NiFi to datadog")
+public class DataDogReportingTask extends AbstractReportingTask {
+
+static final AllowableValue DATADOG_AGENT = new 
AllowableValue("DataDog Agent", "DataDog UDP Agent",
+"Metrics will be sent via locally installed DataDog agent over 
UDP");
--- End diff --

Also it should be made more explicit that this option requires the user 
manually installs an agent and that it is not done automatically.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #655: DataDog support added

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/655#discussion_r72341497
  
--- Diff: 
nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java/org/apache/nifi/reporting/datadog/DDMetricRegistryBuilder.java
 ---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.reporting.datadog;
+
+import com.codahale.metrics.MetricRegistry;
+import org.apache.commons.lang3.RandomStringUtils;
+import org.coursera.metrics.datadog.DatadogReporter;
+import org.coursera.metrics.datadog.transport.HttpTransport;
+import org.coursera.metrics.datadog.transport.Transport;
+import org.coursera.metrics.datadog.transport.UdpTransport;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Class configures MetricRegistry (passed outside or created from 
scratch) with Datadog support
+ */
+public class DDMetricRegistryBuilder {
+
+
+private MetricRegistry metricRegistry = null;
+private String name = null;
+private List tags = Arrays.asList();
+private DatadogReporter datadogReporter;
+private String apiKey = "";
+private Transport transport;
+
+public DDMetricRegistryBuilder setMetricRegistry(MetricRegistry 
metricRegistry) {
+this.metricRegistry = metricRegistry;
+return this;
+}
+
+public DDMetricRegistryBuilder setName(String name) {
+this.name = name;
+return this;
+}
+
+public DDMetricRegistryBuilder setTags(List tags) {
+this.tags = tags;
+return this;
+}
+
+public DatadogReporter getDatadogReporter() {
+return datadogReporter;
+}
+
+public MetricRegistry build(String apiKey) throws IOException {
+if (metricRegistry == null)
+metricRegistry = new MetricRegistry();
+
+if (name == null) {
+name = RandomStringUtils.randomAlphanumeric(8);
--- End diff --

I'm confused where the "name" field comes into play. I don't see it read 
anywhere except to check if it's null here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #655: DataDog support added

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/655#discussion_r72340427
  
--- Diff: 
nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-nar/src/main/resources/META-INF/LICENSE
 ---
@@ -0,0 +1,585 @@
+
+ Apache License
+   Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other 
modifications
+  represent, as a whole, an original work of authorship. For the 
purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces 
of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright 
owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control 
systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  (except as stated in this section) patent license to make, have made,
+  use, offer to sell, sell, import, and otherwise transfer the Work,
+  where such license applies only to those patent

[GitHub] nifi pull request #655: DataDog support added

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/655#discussion_r72340096
  
--- Diff: nifi-assembly/NOTICE ---
@@ -1,4 +1,4 @@
-Apache NiFi
+wApache NiFi
--- End diff --

typo


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394552#comment-15394552
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/563
  
@olegz, when you get a chance could you review the Get/Consume Kafka 
changes in this PR?


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2405) StandardSnippetDAO using incorrect equality check for strings

2016-07-26 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394536#comment-15394536
 ] 

ASF subversion and git services commented on NIFI-2405:
---

Commit 15d1e1bbf83fb8d5259d24997b7c0c56aa8e636e in nifi's branch 
refs/heads/master from [~joewitt]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=15d1e1b ]

NIFI-2405 corrected string equality checks

This closes #723.


> StandardSnippetDAO using incorrect equality check for strings
> -
>
> Key: NIFI-2405
> URL: https://issues.apache.org/jira/browse/NIFI-2405
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Joseph Witt
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> In reviewing the DAOs and other things for NIFI-2339 stumbled across improper 
> string equality checking in StandardSnippetDAO.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2405) StandardSnippetDAO using incorrect equality check for strings

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394539#comment-15394539
 ] 

ASF GitHub Bot commented on NIFI-2405:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/723
  
Looks good and build OK. Merged to master.


> StandardSnippetDAO using incorrect equality check for strings
> -
>
> Key: NIFI-2405
> URL: https://issues.apache.org/jira/browse/NIFI-2405
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Joseph Witt
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> In reviewing the DAOs and other things for NIFI-2339 stumbled across improper 
> string equality checking in StandardSnippetDAO.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2405) StandardSnippetDAO using incorrect equality check for strings

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394537#comment-15394537
 ] 

ASF GitHub Bot commented on NIFI-2405:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/723


> StandardSnippetDAO using incorrect equality check for strings
> -
>
> Key: NIFI-2405
> URL: https://issues.apache.org/jira/browse/NIFI-2405
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Joseph Witt
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> In reviewing the DAOs and other things for NIFI-2339 stumbled across improper 
> string equality checking in StandardSnippetDAO.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #723: NIFI-2405 corrected string equality checks

2016-07-26 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/723


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #563: NIFI-2078, 2363, 2364: External state management. CL...

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72334765
  
--- Diff: 
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/pom.xml ---
@@ -35,6 +35,10 @@
 nifi-utils
 
 
+org.apache.nifi
+nifi-expression-language
--- End diff --

This module didn't already have access to EL?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394524#comment-15394524
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72334765
  
--- Diff: 
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/pom.xml ---
@@ -35,6 +35,10 @@
 nifi-utils
 
 
+org.apache.nifi
+nifi-expression-language
--- End diff --

This module didn't already have access to EL?


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2367) Overlapping text in Cluster pane after error

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394495#comment-15394495
 ] 

ASF GitHub Bot commented on NIFI-2367:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/715
  
Reviewing...


> Overlapping text in Cluster pane after error
> 
>
> Key: NIFI-2367
> URL: https://issues.apache.org/jira/browse/NIFI-2367
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Joseph Percivall
>Assignee: Scott Aslan
> Attachments: Screen Shot 2016-07-22 at 2.42.48 PM.png
>
>
> 3 node secure cluster
> In cluster pane
> hit 500 error when refreshing pane (output in developer tools): 
> https://localhost:8481/nifi-api/controller/cluster Failed to load resource: 
> the server responded with a status of 500 (Internal Server Error)
> See overlapping text of "fix" and "home" in resulting pane (see attached 
> screenshot).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #563: NIFI-2078, 2363, 2364: External state management. CL...

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72330212
  
--- Diff: 
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/ConsumeKafka.java
 ---
@@ -54,7 +72,12 @@
 @InputRequirement(Requirement.INPUT_FORBIDDEN)
 @CapabilityDescription("Consumes messages from Apache Kafka")
 @Tags({ "Kafka", "Get", "Ingest", "Ingress", "Topic", "PubSub", "Consume" 
})
-public class ConsumeKafka extends AbstractKafkaProcessor> {
+@Stateful(scopes = {Scope.EXTERNAL}, description = "After consuming 
messages, ConsumeKafka commits its offset information to Kafka" +
+" so that the state of a consumer group can be retained across 
events such as consumer reconnect." +
+" Offsets can be cleared when there is no consumer subscribing 
with the same consumer group id." +
+" It may take more than 30 seconds for a consumer group to become 
able to be cleared after it is stopped from NiFi." +
+" Once offsets are cleared, ConsumeKafka will resume consuming 
messages based on Offset Reset configuration.")
+public class ConsumeKafka extends AbstractKafkaProcessor> implements ExternalStateManager {
--- End diff --

Since "ExternalStateManager" is something that the developer will need to 
implement if they will use it, this should be documented in the Developer guide


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #724: NIFI-2403 Ensuring uniqueness on user identities and group ...

2016-07-26 Thread mcgilman

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/724
  
@bbende This is a great start. I've verified that we can no longer add new 
users or groups with the same name as existing users or groups. However, I 
think we need to additionally ensure uniqueness between user identities and 
group names. Also, the unique constraint appears to only be checked when 
creating the user or group. Updating still allows the identities to become the 
same.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2403) Users and Groups with same name/identity cause problems in FileAuthorizer

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394482#comment-15394482
 ] 

ASF GitHub Bot commented on NIFI-2403:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/724
  
@bbende This is a great start. I've verified that we can no longer add new 
users or groups with the same name as existing users or groups. However, I 
think we need to additionally ensure uniqueness between user identities and 
group names. Also, the unique constraint appears to only be checked when 
creating the user or group. Updating still allows the identities to become the 
same.


> Users and Groups with same name/identity cause problems in FileAuthorizer
> -
>
> Key: NIFI-2403
> URL: https://issues.apache.org/jira/browse/NIFI-2403
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
> Fix For: 1.0.0
>
>
> Currently the FileAuthorizer does not check for uniqueness of user identities 
> or group names, and this leads to non-deterministic behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #675: LSB Adherence

2016-07-26 Thread PuspenduBanerjee

Github user PuspenduBanerjee commented on a diff in the pull request:

https://github.com/apache/nifi/pull/675#discussion_r72326901
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
 ---
@@ -265,20 +265,54 @@ run() {
 BOOTSTRAP_DIR_PARAMS="${BOOTSTRAP_LOG_PARAMS} ${BOOTSTRAP_PID_PARAMS} 
${BOOTSTRAP_CONF_PARAMS}"
 
 RUN_NIFI_CMD="cd "\""${NIFI_HOME}"\"" && ${sudo_cmd_prefix} 
"\""${JAVA}"\"" -cp "\""${BOOTSTRAP_CLASSPATH}"\"" -Xms12m -Xmx24m 
${BOOTSTRAP_DIR_PARAMS}  org.apache.nifi.bootstrap.RunNiFi"
+LSB_EXIT_STATUS=0
+LSB_NIFI_STATUS_TEMPFILE=`mktemp -t 
LSB_NIFI_STATUS_TEMPFILE.XX`
 
 if [ "$1" = "start" ]; then
 (eval $RUN_NIFI_CMD $@ &)
 else
-(eval $RUN_NIFI_CMD $@)
+(eval $RUN_NIFI_CMD $@ "| tee "$LSB_NIFI_STATUS_TEMPFILE)
 fi
 
 # Wait just a bit (3 secs) to wait for the logging to finish and then 
echo a new-line.
 # We do this to avoid having logs spewed on the console after running 
the command and then not giving
 # control back to the user
 sleep 3
 echo
+if [ "$1" = "status" ]; then
+LSB_EXIT_STATUS= getLSBStatus
+fi
+return $LSB_EXIT_STATUS
 }
 
+# Standard Status codes for LSB compliance
+# See: 
http://refspecs.linuxbase.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html
+# If the status action is requested, the init script will return the 
following exit status codes.
+# 0program is running or service is OK
+# 1program is dead and /var/run pid file exists
+# 2program is dead and /var/lock lock file exists
+# 3program is not running
+# 4program or service status is unknown
+# 5-99 reserved for future LSB use
+# 100-149  reserved for distribution use
+# 150-199  reserved for application use
+# 200-254  reserved
+
+getLSBStatus() {
+  LSB_EXIT_STATUS=4
+  if [ `grep -c "not.*running" ${LSB_NIFI_STATUS_TEMPFILE}` -gt 0 ]; then
--- End diff --

Need to see, if that approach is side-effects free. A short term solution
could be getting it done with log parsing.  A long term solution could be
creating necessary RunNifi companion because some system relies on systemd
and some on upstart.

Thanks & Regards,
Puspendu Banerjee

On Jul 26, 2016 10:19 AM, "Aldrin Piri"  wrote:

> In
> 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
> :
>
> > +# Standard Status codes for LSB compliance
> > +# See: 
http://refspecs.linuxbase.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html
> > +# If the status action is requested, the init script will return the 
following exit status codes.
> > +# 0program is running or service is OK
> > +# 1program is dead and /var/run pid file exists
> > +# 2program is dead and /var/lock lock file exists
> > +# 3program is not running
> > +# 4program or service status is unknown
> > +# 5-99 reserved for future LSB use
> > +# 100-149  reserved for distribution use
> > +# 150-199  reserved for application use
> > +# 200-254  reserved
> > +
> > +getLSBStatus() {
> > +  LSB_EXIT_STATUS=4
> > +  if [ `grep -c "not.*running" ${LSB_NIFI_STATUS_TEMPFILE}` -gt 0 ]; 
then
>
> Thinking through this a bit more, what do you think about the NiFi
> application returning an appropriate status code from RunNiFi with a
> System.exit? This would remove ambiguity and avoid the need to grep for
> specific patterns.
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> 
,
> or mute the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394460#comment-15394460
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72326199
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ClearComponentStateEndpointMerger.java
 ---
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.cluster.coordination.http.endpoints;
+
+import org.apache.nifi.cluster.manager.NodeResponse;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.web.api.entity.ClearComponentStateResultEntity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URI;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+public class ClearComponentStateEndpointMerger extends 
AbstractSingleEntityEndpoint {
+public static final Pattern PROCESSOR_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/processors/[a-f0-9\\-]{36}/state/clear-requests");
+public static final Pattern CONTROLLER_SERVICE_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/controller-services/[a-f0-9\\-]{36}/state/clear-requests");
+public static final Pattern REPORTING_TASK_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/reporting-tasks/[a-f0-9\\-]{36}/state/clear-requests");
--- End diff --

Ah I see that's how it's done in all the classes in the same package


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #563: NIFI-2078, 2363, 2364: External state management. CL...

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72326199
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ClearComponentStateEndpointMerger.java
 ---
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.cluster.coordination.http.endpoints;
+
+import org.apache.nifi.cluster.manager.NodeResponse;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.web.api.entity.ClearComponentStateResultEntity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URI;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+public class ClearComponentStateEndpointMerger extends 
AbstractSingleEntityEndpoint {
+public static final Pattern PROCESSOR_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/processors/[a-f0-9\\-]{36}/state/clear-requests");
+public static final Pattern CONTROLLER_SERVICE_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/controller-services/[a-f0-9\\-]{36}/state/clear-requests");
+public static final Pattern REPORTING_TASK_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/reporting-tasks/[a-f0-9\\-]{36}/state/clear-requests");
--- End diff --

Ah I see that's how it's done in all the classes in the same package


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394457#comment-15394457
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72325932
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/ClusterProtocolHeartbeatMonitor.java
 ---
@@ -91,7 +92,7 @@ public ClusterProtocolHeartbeatMonitor(final 
ClusterCoordinator clusterCoordinat
 this.clusterNodesPath = 
zkClientConfig.resolvePath("cluster/nodes");
 
 String hostname = 
properties.getProperty(NiFiProperties.CLUSTER_NODE_ADDRESS);
-if (hostname == null) {
+if (StringUtils.isEmpty(hostname)) {
--- End diff --

This change will conflict with @markap14 change in this PR: 
https://github.com/apache/nifi/pull/688/files


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #563: NIFI-2078, 2363, 2364: External state management. CL...

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72325932
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/ClusterProtocolHeartbeatMonitor.java
 ---
@@ -91,7 +92,7 @@ public ClusterProtocolHeartbeatMonitor(final 
ClusterCoordinator clusterCoordinat
 this.clusterNodesPath = 
zkClientConfig.resolvePath("cluster/nodes");
 
 String hostname = 
properties.getProperty(NiFiProperties.CLUSTER_NODE_ADDRESS);
-if (hostname == null) {
+if (StringUtils.isEmpty(hostname)) {
--- End diff --

This change will conflict with @markap14 change in this PR: 
https://github.com/apache/nifi/pull/688/files


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394454#comment-15394454
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72325489
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/components/state/ExternalStateManager.java
 ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.components.state;
+
+import org.apache.nifi.annotation.behavior.Stateful;
+
+import java.io.IOException;
+
+/**
+ * 
+ * The ExternalStateManager is responsible for providing NiFi a mechanism 
for retrieving
+ * and clearing state stored in an external system a NiFi component 
interact with.
+ * 
+ *
+ * 
+ * When calling methods in this class, the state is always 
retrieved/cleared from external system
+ * regardless NiFi instance is a part of a cluster or standalone.
+ * 
+ *
+ * 
+ * This mechanism is designed to allow developers to easily store and 
retrieve small amounts of state.
+ * Since implementation of this interface interacts with remote system, 
one should consider the cost of
+ * retrieving this data, and the amount of data should be kept to the 
minimum required.
--- End diff --

This paragraph doesn't make sense for External state since the user 
probably isn't choosing what to store


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #563: NIFI-2078, 2363, 2364: External state management. CL...

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72325489
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/components/state/ExternalStateManager.java
 ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.components.state;
+
+import org.apache.nifi.annotation.behavior.Stateful;
+
+import java.io.IOException;
+
+/**
+ * 
+ * The ExternalStateManager is responsible for providing NiFi a mechanism 
for retrieving
+ * and clearing state stored in an external system a NiFi component 
interact with.
+ * 
+ *
+ * 
+ * When calling methods in this class, the state is always 
retrieved/cleared from external system
+ * regardless NiFi instance is a part of a cluster or standalone.
+ * 
+ *
+ * 
+ * This mechanism is designed to allow developers to easily store and 
retrieve small amounts of state.
+ * Since implementation of this interface interacts with remote system, 
one should consider the cost of
+ * retrieving this data, and the amount of data should be kept to the 
minimum required.
--- End diff --

This paragraph doesn't make sense for External state since the user 
probably isn't choosing what to store


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2339) Component details need to be removed from all error messages

2016-07-26 Thread Joseph Witt (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394395#comment-15394395
 ] 

Joseph Witt commented on NIFI-2339:
---

[~mcgilman] Matt i don't know if this is all you had in mind but I did a fairly 
thorough review through the codebase searching through the 
- DTOs
- DAOs
- Things which created exceptions like IllegalArg/State, NPE, CoreNiFiException
- Things which use getName, getType, etc..

And all the uses of internal details which could be authorization leakage in 
nature for exceptions all seemed quite good (using identifiers only).  So just 
these few tweaks seemed to be all that is needed.

If you could take a look that would be great.

> Component details need to be removed from all error messages
> 
>
> Key: NIFI-2339
> URL: https://issues.apache.org/jira/browse/NIFI-2339
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Joseph Witt
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Error and informative message often contain component details which need to 
> be scrubbed when the user is not authorized for reading that component.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2403) Users and Groups with same name/identity cause problems in FileAuthorizer

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394392#comment-15394392
 ] 

ASF GitHub Bot commented on NIFI-2403:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/724
  
Reviewing


> Users and Groups with same name/identity cause problems in FileAuthorizer
> -
>
> Key: NIFI-2403
> URL: https://issues.apache.org/jira/browse/NIFI-2403
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
> Fix For: 1.0.0
>
>
> Currently the FileAuthorizer does not check for uniqueness of user identities 
> or group names, and this leads to non-deterministic behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #724: NIFI-2403 Ensuring uniqueness on user identities and group ...

2016-07-26 Thread mcgilman

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/724
  
Reviewing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2339) Component details need to be removed from all error messages

2016-07-26 Thread Joseph Witt (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2339:
--
Status: Patch Available  (was: Open)

> Component details need to be removed from all error messages
> 
>
> Key: NIFI-2339
> URL: https://issues.apache.org/jira/browse/NIFI-2339
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Joseph Witt
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Error and informative message often contain component details which need to 
> be scrubbed when the user is not authorized for reading that component.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2389) Legacy authorized-users.xml and normalized users

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394388#comment-15394388
 ] 

ASF GitHub Bot commented on NIFI-2389:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/719
  
Thanks @bbende... looks good. This has been merged to master.


> Legacy authorized-users.xml and normalized users
> 
>
> Key: NIFI-2389
> URL: https://issues.apache.org/jira/browse/NIFI-2389
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Gilman
>Assignee: Bryan Bende
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Ensure user identity normalization rules are applied when converting a legacy 
> authorized users file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2339) Component details need to be removed from all error messages

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394387#comment-15394387
 ] 

ASF GitHub Bot commented on NIFI-2339:
--

GitHub user joewitt opened a pull request:

https://github.com/apache/nifi/pull/726

NIFI-2339 Removed some outputs which could cause errors to reveal 
unauthorized data



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/joewitt/incubator-nifi NIFI-2339

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/726.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #726


commit 01a07818062419e52353bf0ed6a3309f6ad3f66d
Author: joewitt 
Date:   2016-07-26T19:25:49Z

NIFI-2339 Removed some outputs which could cause errors to reveal 
unauthorized data.




> Component details need to be removed from all error messages
> 
>
> Key: NIFI-2339
> URL: https://issues.apache.org/jira/browse/NIFI-2339
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Joseph Witt
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Error and informative message often contain component details which need to 
> be scrubbed when the user is not authorized for reading that component.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #719: NIFI-2389 Refactoring identity mapping and applying it to F...

2016-07-26 Thread mcgilman

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/719
  
Thanks @bbende... looks good. This has been merged to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #726: NIFI-2339 Removed some outputs which could cause err...

2016-07-26 Thread joewitt

GitHub user joewitt opened a pull request:

https://github.com/apache/nifi/pull/726

NIFI-2339 Removed some outputs which could cause errors to reveal 
unauthorized data



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/joewitt/incubator-nifi NIFI-2339

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/726.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #726


commit 01a07818062419e52353bf0ed6a3309f6ad3f66d
Author: joewitt 
Date:   2016-07-26T19:25:49Z

NIFI-2339 Removed some outputs which could cause errors to reveal 
unauthorized data.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2389) Legacy authorized-users.xml and normalized users

2016-07-26 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394380#comment-15394380
 ] 

ASF subversion and git services commented on NIFI-2389:
---

Commit c3b4872b552922058c6536e0810dabfe9ca20b10 in nifi's branch 
refs/heads/master from [~bbende]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=c3b4872 ]

NIFI-2389 Refactoring identity mapping and applying it to FileAuthorizer for 
initial admin, cluster nodes, and legacy authorized users. This closes #719


> Legacy authorized-users.xml and normalized users
> 
>
> Key: NIFI-2389
> URL: https://issues.apache.org/jira/browse/NIFI-2389
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Gilman
>Assignee: Bryan Bende
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Ensure user identity normalization rules are applied when converting a legacy 
> authorized users file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #719: NIFI-2389 Refactoring identity mapping and applying ...

2016-07-26 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/719


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394363#comment-15394363
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user brosander commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72317301
  
--- Diff: nifi-assembly/pom.xml ---
@@ -363,147 +363,6 @@ language governing permissions and limitations under 
the License. -->
 
 
 
-
--- End diff --

@JPercivall I moved them into the pom of nifi-resources so that the 
filtering could happen before the files are packaged.  This kept me from 
needing to duplicate them so that the toolkit could bundle a valid 
nifi.properties, instead I can just resolve an up to date one.


> Command Line Keystore and Truststore utility
> 
>
> Key: NIFI-2193
> URL: https://issues.apache.org/jira/browse/NIFI-2193
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Bryan Rosander
>Assignee: Bryan Rosander
>
> In order to facilitate secure setup of NiFi, it would be useful to have a 
> command line utility capable of generating the required keystores, 
> truststore, and relevant configuration files.
> It should be able to generate keystores for each NiFi node, a truststore that 
> they all use, and relevant passwords and configuration files for using the 
> keystores and truststore.
> Additionally, in order to support distributed deployment, a web based 
> certificate authority with corresponding client will allow for each NiFi 
> instance to generate its own keypair and then request signing by the CA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

2016-07-26 Thread brosander

Github user brosander commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72317301
  
--- Diff: nifi-assembly/pom.xml ---
@@ -363,147 +363,6 @@ language governing permissions and limitations under 
the License. -->
 
 
 
-
--- End diff --

@JPercivall I moved them into the pom of nifi-resources so that the 
filtering could happen before the files are packaged.  This kept me from 
needing to duplicate them so that the toolkit could bundle a valid 
nifi.properties, instead I can just resolve an up to date one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394344#comment-15394344
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72315293
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ClearComponentStateEndpointMerger.java
 ---
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.cluster.coordination.http.endpoints;
+
+import org.apache.nifi.cluster.manager.NodeResponse;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.web.api.entity.ClearComponentStateResultEntity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URI;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+public class ClearComponentStateEndpointMerger extends 
AbstractSingleEntityEndpoint {
+public static final Pattern PROCESSOR_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/processors/[a-f0-9\\-]{36}/state/clear-requests");
+public static final Pattern CONTROLLER_SERVICE_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/controller-services/[a-f0-9\\-]{36}/state/clear-requests");
+public static final Pattern REPORTING_TASK_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/reporting-tasks/[a-f0-9\\-]{36}/state/clear-requests");
--- End diff --

Hard coding these URI patterns feels wrong. Any changes to these URIs in 
another PR could mess this up entirely without anyone noticing.


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394323#comment-15394323
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72313026
  
--- Diff: nifi-assembly/pom.xml ---
@@ -363,147 +363,6 @@ language governing permissions and limitations under 
the License. -->
 
 
 
-
--- End diff --

Why are all of these properties being removed from the assembly pom?


> Command Line Keystore and Truststore utility
> 
>
> Key: NIFI-2193
> URL: https://issues.apache.org/jira/browse/NIFI-2193
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Bryan Rosander
>Assignee: Bryan Rosander
>
> In order to facilitate secure setup of NiFi, it would be useful to have a 
> command line utility capable of generating the required keystores, 
> truststore, and relevant configuration files.
> It should be able to generate keystores for each NiFi node, a truststore that 
> they all use, and relevant passwords and configuration files for using the 
> keystores and truststore.
> Additionally, in order to support distributed deployment, a web based 
> certificate authority with corresponding client will allow for each NiFi 
> instance to generate its own keypair and then request signing by the CA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

2016-07-26 Thread JPercivall

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72313026
  
--- Diff: nifi-assembly/pom.xml ---
@@ -363,147 +363,6 @@ language governing permissions and limitations under 
the License. -->
 
 
 
-
--- End diff --

Why are all of these properties being removed from the assembly pom?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2318) Node Events shown in cluster table often indicate that the node's status changed to CONNECTED many times

2016-07-26 Thread Joseph Percivall (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall updated NIFI-2318:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Node Events shown in cluster table often indicate that the node's status 
> changed to CONNECTED many times
> 
>
> Key: NIFI-2318
> URL: https://issues.apache.org/jira/browse/NIFI-2318
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Minor
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2316) After restarting cluster, all nodes sometimes come up with all components stopped

2016-07-26 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394294#comment-15394294
 ] 

ASF subversion and git services commented on NIFI-2316:
---

Commit 52bc23f5dbcc3a236df1fa88744c277e5d60e244 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=52bc23f ]

NIFI-2316, NIFI-2318: Ensure that we do not save the flow before initializing 
the Run Status of components. Clarify the Node Event messages

This closes #678

Signed-off-by: jpercivall 


> After restarting cluster, all nodes sometimes come up with all components 
> stopped
> -
>
> Key: NIFI-2316
> URL: https://issues.apache.org/jira/browse/NIFI-2316
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> When I restart an entire cluster, I see sometimes that all processors, etc. 
> are stopped. Sometimes this can also result in some nodes not being able to 
> join the cluster if they have a root group port, as the run status is not the 
> same across the cluster, so the new node cannot inherit the flow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2318) Node Events shown in cluster table often indicate that the node's status changed to CONNECTED many times

2016-07-26 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394295#comment-15394295
 ] 

ASF subversion and git services commented on NIFI-2318:
---

Commit 52bc23f5dbcc3a236df1fa88744c277e5d60e244 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=52bc23f ]

NIFI-2316, NIFI-2318: Ensure that we do not save the flow before initializing 
the Run Status of components. Clarify the Node Event messages

This closes #678

Signed-off-by: jpercivall 


> Node Events shown in cluster table often indicate that the node's status 
> changed to CONNECTED many times
> 
>
> Key: NIFI-2318
> URL: https://issues.apache.org/jira/browse/NIFI-2318
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Minor
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2316) After restarting cluster, all nodes sometimes come up with all components stopped

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394297#comment-15394297
 ] 

ASF GitHub Bot commented on NIFI-2316:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/678


> After restarting cluster, all nodes sometimes come up with all components 
> stopped
> -
>
> Key: NIFI-2316
> URL: https://issues.apache.org/jira/browse/NIFI-2316
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> When I restart an entire cluster, I see sometimes that all processors, etc. 
> are stopped. Sometimes this can also result in some nodes not being able to 
> join the cluster if they have a root group port, as the run status is not the 
> same across the cluster, so the new node cannot inherit the flow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #678: NIFI-2316, NIFI-2318: Ensure that we do not save the...

2016-07-26 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/678


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2389) Legacy authorized-users.xml and normalized users

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394272#comment-15394272
 ] 

ASF GitHub Bot commented on NIFI-2389:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/719
  
Reviewing...


> Legacy authorized-users.xml and normalized users
> 
>
> Key: NIFI-2389
> URL: https://issues.apache.org/jira/browse/NIFI-2389
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Gilman
>Assignee: Bryan Bende
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Ensure user identity normalization rules are applied when converting a legacy 
> authorized users file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #719: NIFI-2389 Refactoring identity mapping and applying it to F...

2016-07-26 Thread mcgilman

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/719
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-2408) If a node is disconnected, it should still provide user ability to see the 'cluster screen'

2016-07-26 Thread Mark Payne (JIRA)

Mark Payne created NIFI-2408:


 Summary: If a node is disconnected, it should still provide user 
ability to see the 'cluster screen'
 Key: NIFI-2408
 URL: https://issues.apache.org/jira/browse/NIFI-2408
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core UI
Reporter: Mark Payne
 Fix For: 1.0.0


A decision was made a while back that if a node is disconnected from the 
cluster, the 'cluster screen' should not be available in the menu in the 
top-right corner. However, after using the app like this, I now think it was a 
bad decision. The node is still part of the cluster, even though it is 
disconnected. Unless the user clicks 'remove', the node is a part of the 
cluster, so should be able to render the cluster information.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2316) After restarting cluster, all nodes sometimes come up with all components stopped

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394267#comment-15394267
 ] 

ASF GitHub Bot commented on NIFI-2316:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/678
  
+1

Visually verified code and did a contrib check build. In a 3 node secure 
cluster, I did a lot of restarting, starting/stopping processors, disconnecting 
nodes and trying to hit corner cases and it all functioned as expected. Thanks 
@markap14, I will merge it in.


> After restarting cluster, all nodes sometimes come up with all components 
> stopped
> -
>
> Key: NIFI-2316
> URL: https://issues.apache.org/jira/browse/NIFI-2316
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> When I restart an entire cluster, I see sometimes that all processors, etc. 
> are stopped. Sometimes this can also result in some nodes not being able to 
> join the cluster if they have a root group port, as the run status is not the 
> same across the cluster, so the new node cannot inherit the flow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #678: NIFI-2316, NIFI-2318: Ensure that we do not save the flow b...

2016-07-26 Thread JPercivall

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/678
  
+1

Visually verified code and did a contrib check build. In a 3 node secure 
cluster, I did a lot of restarting, starting/stopping processors, disconnecting 
nodes and trying to hit corner cases and it all functioned as expected. Thanks 
@markap14, I will merge it in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2296) The hover text for the Search icon in the Upload Template window is confusing

2016-07-26 Thread Matt Gilman (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-2296:
--
Status: Patch Available  (was: In Progress)

> The hover text for the Search icon in the Upload Template window is confusing
> -
>
> Key: NIFI-2296
> URL: https://issues.apache.org/jira/browse/NIFI-2296
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Matt Gilman
>Priority: Minor
>  Labels: UI
> Attachments: NIFI-2296_noFileChosen.png, NIFI-2296_xmlChosen.png
>
>
> When the user hovers over the Search icon in the Upload Template window, the 
> text displayed is either "No file chosen" or the name of the template xml 
> chosen.  This text doesn't aide the user in determining what the icon/button 
> is for and also displays redundant information when a template has been 
> selected.  See attached screenshots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2338) Replay Authorization Check happening twice

2016-07-26 Thread Mark Payne (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-2338:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Replay Authorization Check happening twice
> --
>
> Key: NIFI-2338
> URL: https://issues.apache.org/jira/browse/NIFI-2338
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Verified replay authorization and ensure it's only executed when 
> appropriately. Currently, its happening in the framework and in the 
> provenance repository.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2346) Add resource for authorizing all data access

2016-07-26 Thread Mark Payne (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-2346:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Add resource for authorizing all data access
> 
>
> Key: NIFI-2346
> URL: https://issues.apache.org/jira/browse/NIFI-2346
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework, Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Blocker
> Fix For: 1.0.0
>
>
> - Data resource should be used to authorize the provenance events and flow 
> files during queues.
> - Data resource (and data transfer resource) requires entire proxy chain 
> authorization.
> - Flow file queue will be driven by the source of the connection.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2346) Add resource for authorizing all data access

2016-07-26 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394250#comment-15394250
 ] 

ASF subversion and git services commented on NIFI-2346:
---

Commit 69586d8bd00b0e1e6cc606411363851b75684554 in nifi's branch 
refs/heads/master from [~mcgilman]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=69586d8 ]

NIFI-2346:
- Introducing data resource for authorizing provenance events and queue listing.
- Authorizing entire proxy chain for data resource and data transfer resource.
NIFI-2338:
- Ensuring that replay authorization only happens once.

- Allowing users with access to policies for a component to be able to access 
all policies for that component.
-- Includes the component, data, data transfers, and policies.
- Fixing drop request completion to update the correct queued field.
- Fixing access control check for listing and emptying queues.
- Reseting selected policy when re-opening the policy management page.
- Fixing button/link visibility for available actions in policy management page.
- Fixing policy issues with policy removal when the underlying component is 
deleted.
- Updating file authorizer seeding to grant data access to node's in the 
cluster.

This closes #720.


> Add resource for authorizing all data access
> 
>
> Key: NIFI-2346
> URL: https://issues.apache.org/jira/browse/NIFI-2346
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework, Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Blocker
> Fix For: 1.0.0
>
>
> - Data resource should be used to authorize the provenance events and flow 
> files during queues.
> - Data resource (and data transfer resource) requires entire proxy chain 
> authorization.
> - Flow file queue will be driven by the source of the connection.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2338) Replay Authorization Check happening twice

2016-07-26 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394251#comment-15394251
 ] 

ASF subversion and git services commented on NIFI-2338:
---

Commit 69586d8bd00b0e1e6cc606411363851b75684554 in nifi's branch 
refs/heads/master from [~mcgilman]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=69586d8 ]

NIFI-2346:
- Introducing data resource for authorizing provenance events and queue listing.
- Authorizing entire proxy chain for data resource and data transfer resource.
NIFI-2338:
- Ensuring that replay authorization only happens once.

- Allowing users with access to policies for a component to be able to access 
all policies for that component.
-- Includes the component, data, data transfers, and policies.
- Fixing drop request completion to update the correct queued field.
- Fixing access control check for listing and emptying queues.
- Reseting selected policy when re-opening the policy management page.
- Fixing button/link visibility for available actions in policy management page.
- Fixing policy issues with policy removal when the underlying component is 
deleted.
- Updating file authorizer seeding to grant data access to node's in the 
cluster.

This closes #720.


> Replay Authorization Check happening twice
> --
>
> Key: NIFI-2338
> URL: https://issues.apache.org/jira/browse/NIFI-2338
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Verified replay authorization and ensure it's only executed when 
> appropriately. Currently, its happening in the framework and in the 
> provenance repository.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #720: Introduce data resource for authorizing Provenance and Queu...

2016-07-26 Thread markap14

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/720
  
Code looks good now. Unit tests are good. Tested functionality and all 
appears to work as expected. +1 merged to master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #720: Introduce data resource for authorizing Provenance a...

2016-07-26 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/720


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2176) Able to create a template without a name

2016-07-26 Thread Matt Gilman (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-2176:
--
Status: Patch Available  (was: Reopened)

> Able to create a template without a name
> 
>
> Key: NIFI-2176
> URL: https://issues.apache.org/jira/browse/NIFI-2176
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Matt Gilman
>  Labels: UI
> Fix For: 1.0.0
>
>
> Select "Create Template" from the Operator palette.  Immediately select 
> "Create" from the "Create Template" dialog leaving the name field blank.  A 
> success message "Template '' was successfully created." is displayed.
> In 0.x, if a user attempted this they would get an error "Template name 
> cannot be blank."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (NIFI-2176) Able to create a template without a name

2016-07-26 Thread Matt Gilman (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman reopened NIFI-2176:
---

> Able to create a template without a name
> 
>
> Key: NIFI-2176
> URL: https://issues.apache.org/jira/browse/NIFI-2176
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Matt Gilman
>  Labels: UI
> Fix For: 1.0.0
>
>
> Select "Create Template" from the Operator palette.  Immediately select 
> "Create" from the "Create Template" dialog leaving the name field blank.  A 
> success message "Template '' was successfully created." is displayed.
> In 0.x, if a user attempted this they would get an error "Template name 
> cannot be blank."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2224) Upload template dialog does not show template name selected

2016-07-26 Thread Matt Gilman (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-2224:
--
Status: Patch Available  (was: In Progress)

> Upload template dialog does not show template name selected
> ---
>
> Key: NIFI-2224
> URL: https://issues.apache.org/jira/browse/NIFI-2224
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Affects Versions: 1.0.0
> Environment: Mac OS X; Chrome
>Reporter: Andrew Lim
>Assignee: Matt Gilman
>  Labels: UI
> Fix For: 1.0.0
>
> Attachments: NIFI-2224.mp4
>
>
> Following STRs are shown in the attached video:
> 1. Select Upload Template from operator palette
> 2. Select magnifying glass icon to browse for template
> 3. Select template '1.xml'
> 4. Select open button
> 5. Upload template dialog in NiFi shows '1.xml' as selected
> 6. Select Upload button
> 7. Template uploaded successfully
> 8. Select Upload Template from operator palette
> 9. Select magnifying glass icon to browse for template
> 10 Select template '1.xml' AGAIN
> 11. Select open button
> 12. Upload template dialog in NiFi does NOT show '1.xml' as selected
> Separate bug (NIFI-2223)
> 13.  Select Upload button
> 14. Template uploaded successfully



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2223) Able to upload same template multiple times

2016-07-26 Thread Matt Gilman (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-2223:
--
Status: Patch Available  (was: In Progress)

> Able to upload same template multiple times
> ---
>
> Key: NIFI-2223
> URL: https://issues.apache.org/jira/browse/NIFI-2223
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Affects Versions: 1.0.0
> Environment: Mac OS X; Chrome
>Reporter: Andrew Lim
>Assignee: Matt Gilman
>  Labels: UI
> Fix For: 1.0.0
>
> Attachments: NIFI-2223_ErrorMessage.png, NIFI-2223_sameTemp3X.png
>
>
> UI allows the same template to be uploaded multiple times as shown in the 
> attached screenshot.
> In 0.x, this was not allowed and the error message "A templated named 'XYZ' 
> already exists."  Screenshot attached for this also.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2316) After restarting cluster, all nodes sometimes come up with all components stopped

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394211#comment-15394211
 ] 

ASF GitHub Bot commented on NIFI-2316:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/678
  
For tracking, I encountered this issue[1] while testing but I believe it's 
unrelated to this PR. I will continue reviewing.

[1] https://issues.apache.org/jira/browse/NIFI-2406


> After restarting cluster, all nodes sometimes come up with all components 
> stopped
> -
>
> Key: NIFI-2316
> URL: https://issues.apache.org/jira/browse/NIFI-2316
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> When I restart an entire cluster, I see sometimes that all processors, etc. 
> are stopped. Sometimes this can also result in some nodes not being able to 
> join the cluster if they have a root group port, as the run status is not the 
> same across the cluster, so the new node cannot inherit the flow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2406) Rare start-up problems resulting in all nodes disconnected

2016-07-26 Thread Joseph Percivall (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall updated NIFI-2406:
---
Attachment: logs.tar.gz

Logs from the 3 node secure cluster that I encountered the issue

> Rare start-up problems resulting in all nodes disconnected
> --
>
> Key: NIFI-2406
> URL: https://issues.apache.org/jira/browse/NIFI-2406
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Percivall
> Attachments: logs.tar.gz
>
>
> While testing PR 678[1], I came across a time where all the nodes were in a 
> disconnected state and each were in a weird state of heartbeating but not 
> connected.
> Also in the logs there were ~1000 lines of:
> 2016-07-26 11:38:07,841 INFO [Leader Election Notification Thread-1] 
> o.a.n.c.l.e.CuratorLeaderElectionManager 
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@24fae8c6
>  This node has been elected Leader for Role 'Cluster Coordinator'
> This message only gets called here[2] which is a call back for ZK. Also there 
> were many log messages of:
> 2016-07-26 11:54:07,910 WARN [Clustering Tasks Thread-1] 
> o.a.n.c.c.node.NodeClusterCoordinator Failed to determine which node is 
> elected active Cluster Coordinator: ZooKeeper reports the address as 
> localhost:6001, but there is no node with this address
> I believe this is a problem with ZK/NiFi that existed before this PR and not 
> directly related to the PR being reviewed. I will attach a tar of the 3 
> node's logs.
> [1] https://github.com/apache/nifi/pull/678
> [2] 
> https://github.com/apache/nifi/blame/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/leader/election/CuratorLeaderElectionManager.java#L220-L220



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2224) Upload template dialog does not show template name selected

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394106#comment-15394106
 ] 

ASF GitHub Bot commented on NIFI-2224:
--

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/725

Addressing issues surrounding Template uploading

NIFI-2224:
- Ensuring the template form is reset when the upload template dialog is 
closed.

NIFI-2175:
- Not submitting the template form is a template isn't choosen.

NIFI-2176:
- Ensuring a template is specified during creation.

NIFI-2223:
- Ensuring templates with the same name cannot be added.

NIFI-2296:
- Updating the tooltip for the upload template browse button.

- Cleaning up un-used parameters to REST endpoints.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-2175

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/725.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #725


commit 32a6482662add3d41307661d93de06261054e5ff
Author: Matt Gilman 
Date:   2016-07-26T15:02:04Z

NIFI-2224:
- Ensuring the template form is reset when the upload template dialog is 
closed.
NIFI-2175:
- Not submitting the template form is a template isn't choosen.
NIFI-2176:
- Ensuring a template is specified during creation.
NIFI-2223:
- Ensuring templates with the same name cannot be added.
NIFI-2296:
- Updating the tooltip for the upload template browse button.

- Cleaning up un-used parameters to REST endpoints.




> Upload template dialog does not show template name selected
> ---
>
> Key: NIFI-2224
> URL: https://issues.apache.org/jira/browse/NIFI-2224
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Affects Versions: 1.0.0
> Environment: Mac OS X; Chrome
>Reporter: Andrew Lim
>Assignee: Matt Gilman
>  Labels: UI
> Fix For: 1.0.0
>
> Attachments: NIFI-2224.mp4
>
>
> Following STRs are shown in the attached video:
> 1. Select Upload Template from operator palette
> 2. Select magnifying glass icon to browse for template
> 3. Select template '1.xml'
> 4. Select open button
> 5. Upload template dialog in NiFi shows '1.xml' as selected
> 6. Select Upload button
> 7. Template uploaded successfully
> 8. Select Upload Template from operator palette
> 9. Select magnifying glass icon to browse for template
> 10 Select template '1.xml' AGAIN
> 11. Select open button
> 12. Upload template dialog in NiFi does NOT show '1.xml' as selected
> Separate bug (NIFI-2223)
> 13.  Select Upload button
> 14. Template uploaded successfully



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #725: Addressing issues surrounding Template uploading

2016-07-26 Thread mcgilman

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/725

Addressing issues surrounding Template uploading

NIFI-2224:
- Ensuring the template form is reset when the upload template dialog is 
closed.

NIFI-2175:
- Not submitting the template form is a template isn't choosen.

NIFI-2176:
- Ensuring a template is specified during creation.

NIFI-2223:
- Ensuring templates with the same name cannot be added.

NIFI-2296:
- Updating the tooltip for the upload template browse button.

- Cleaning up un-used parameters to REST endpoints.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-2175

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/725.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #725


commit 32a6482662add3d41307661d93de06261054e5ff
Author: Matt Gilman 
Date:   2016-07-26T15:02:04Z

NIFI-2224:
- Ensuring the template form is reset when the upload template dialog is 
closed.
NIFI-2175:
- Not submitting the template form is a template isn't choosen.
NIFI-2176:
- Ensuring a template is specified during creation.
NIFI-2223:
- Ensuring templates with the same name cannot be added.
NIFI-2296:
- Updating the tooltip for the upload template browse button.

- Cleaning up un-used parameters to REST endpoints.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2403) Users and Groups with same name/identity cause problems in FileAuthorizer

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394068#comment-15394068
 ] 

ASF GitHub Bot commented on NIFI-2403:
--

GitHub user bbende opened a pull request:

https://github.com/apache/nifi/pull/724

NIFI-2403 Ensuring uniqueness on user identities and group names



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbende/nifi NIFI-2403

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/724.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #724


commit e632cd337acbc031ae1e722905ae2560cd611cac
Author: Bryan Bende 
Date:   2016-07-26T15:37:17Z

NIFI-2403 Ensuring uniqueness on user identities and group names




> Users and Groups with same name/identity cause problems in FileAuthorizer
> -
>
> Key: NIFI-2403
> URL: https://issues.apache.org/jira/browse/NIFI-2403
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
> Fix For: 1.0.0
>
>
> Currently the FileAuthorizer does not check for uniqueness of user identities 
> or group names, and this leads to non-deterministic behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2347) Pasted Components should inherit policies from the original, not the group it's pasted into

2016-07-26 Thread Matt Gilman (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-2347:
--
Fix Version/s: 1.0.0

> Pasted Components should inherit policies from the original, not the group 
> it's pasted into
> ---
>
> Key: NIFI-2347
> URL: https://issues.apache.org/jira/browse/NIFI-2347
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Percivall
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Copy a processor that you have view permissions but not write
> Paste it into a group that you have full permissions
> Pasted processor inherits from the group and gives you full permissions.
> Instead it should copy/inherit the policies from the copied component.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-2347) Pasted Components should inherit policies from the original, not the group it's pasted into

2016-07-26 Thread Matt Gilman (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman reassigned NIFI-2347:
-

Assignee: Matt Gilman

> Pasted Components should inherit policies from the original, not the group 
> it's pasted into
> ---
>
> Key: NIFI-2347
> URL: https://issues.apache.org/jira/browse/NIFI-2347
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Percivall
>Assignee: Matt Gilman
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Copy a processor that you have view permissions but not write
> Paste it into a group that you have full permissions
> Pasted processor inherits from the group and gives you full permissions.
> Instead it should copy/inherit the policies from the copied component.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-2151) Property edit dialog when resized very small results in OK button covering resize widget

2016-07-26 Thread Scott Aslan (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan reassigned NIFI-2151:
-

Assignee: Scott Aslan

> Property edit dialog when resized very small results in OK button covering 
> resize widget
> 
>
> Key: NIFI-2151
> URL: https://issues.apache.org/jira/browse/NIFI-2151
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Jeff Storck
>Assignee: Scott Aslan
>Priority: Minor
>
> In the property editing dialog, if it is resized very small, the layout of 
> the OK and cancel buttons get misaligned, and the OK button ends up covering 
> the resize widget.  If there's no text in the box, a very small part of the 
> widget is still clickable and usable.  Once there's text in the box, the user 
> is unable to click on the resize widget. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-2206) Wrong mouse cursor when hovering over 'Cancel' button in Edit Property dialog

2016-07-26 Thread Scott Aslan (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan reassigned NIFI-2206:
-

Assignee: Scott Aslan

> Wrong mouse cursor when hovering over 'Cancel' button in Edit Property dialog
> -
>
> Key: NIFI-2206
> URL: https://issues.apache.org/jira/browse/NIFI-2206
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Scott Aslan
> Fix For: 1.0.0
>
>
> I added a ListFile Processor. Clicked Configure, switched to Properties tab. 
> Clicked to change the value of the "Input Directory" property. When I placed 
> the mouse over the "Cancel" button, the mouse cursor changes to the 'Move' 
> cursor instead of the 'pointer' cursor



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-2404) In the node details pop-up of the cluster panel, there are multiple scroll bars when node events get too long

2016-07-26 Thread Scott Aslan (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan reassigned NIFI-2404:
-

Assignee: Scott Aslan

> In the node details pop-up of the cluster panel, there are multiple scroll 
> bars when node events get too long
> -
>
> Key: NIFI-2404
> URL: https://issues.apache.org/jira/browse/NIFI-2404
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Joseph Percivall
>Assignee: Scott Aslan
>Priority: Trivial
> Attachments: Screen Shot 2016-07-26 at 11.28.43 AM.png
>
>
> Depending on the size of the window, there is sometimes a second scroll bar 
> on the node details pop-up scrolling all the contents when there is plenty of 
> room for the whole pop-up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2028) Site-to-Site Transit URI is inconsistent

2016-07-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393979#comment-15393979
 ] 

ASF GitHub Bot commented on NIFI-2028:
--

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/529
  
Hi @markap14 Updated the PR.

- For HTTP Site-to-Site, simply use Communicant url as transit uri.
- For RAW Site-to-Site, added CustomTransitUriTransaction to keep using 
existing format.

Removed TRANSIT_URI HTTP header since transit url can be retrieved from 
HTTP request object at server side. Also, removed url from 
CommunicationsSession.

I hope this addresses your feedback. Please take a look at it when you have 
time. Thank you!



> Site-to-Site Transit URI is inconsistent
> 
>
> Key: NIFI-2028
> URL: https://issues.apache.org/jira/browse/NIFI-2028
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
> Attachments: send-to-the-same-box.png
>
>
> Site-to-Site client and server create provenance event at both end, and those 
> have Transit URI in it to record how flow files are transferred. However, the 
> URI formats are inconsistent among RAW vs HTTP.
> Test result as follows:
> - These port numbers are configurable in nifi.properties
> - 3080: Web API port (nifi.web.http.port)
> - 3081: Site-to-Site RAW Socket port (nifi.remote.input.socket.port)
> *Before Fix*
> PUSH - RAW
> |Client - SEND: | nifi://localhost:3081/flow-file-uuid |
> |Server - RECEIVE: | nifi://localhost:3081/flow-file-uuid|
> PULL - RAW
> |Client - RECEIVE:| nifi://localhost:3081flow-file-uuid|
> |Server - SEND:   | nifi://localhost:3081/flow-file-uuid|
> PUSH - HTTP
> |Client - SEND:   | http://127.0.0.1:3080/nifi-api/flow-file-uuid|
> |Server - RECEIVE:| nifi://127.0.0.1:57390|
> PULL - HTTP
> |Client - RECEIVE:| http://127.0.0.1:3080/flow-file-uuid|
> |Server - SEND:   | nifi://127.0.0.1:57673|
> *Issues*
> - PULL - RAW, Client - RECEIVE: lacks '/' in between port and flow-file uuid
> - RAW uses server's host and port on both end (by transit url prefix), HTTP 
> should follow this rule
> - HTTP transit uri looks like REST endpoint but it is not a real endpoint. It 
> should be an actual endpoint URI
> - RAW uses hostname, while HTTP uses IP address
> *After Fix*
> PUSH - RAW
> |Client - SEND: | nifi://localhost:3081/flow-file-uuid |
> |Server - RECEIVE: | nifi://localhost:3081/flow-file-uuid|
> PULL - RAW
> |Client - RECEIVE: | nifi://localhost:3081/flow-file-uuid|
> |Server - SEND:   | nifi://localhost:3081/flow-file-uuid|
> PUSH - HTTP
> |Client - SEND:   | 
> http://localhost:3080/nifi-api/data-transfer/input-ports/$port-id/transactions/$tx-id/flow-files|
> |Server - RECEIVE:| 
> http://localhost:3080/nifi-api/data-transfer/input-ports/$port-id/transactions/$tx-id/flow-files|
> PULL - HTTP
> |Client - RECEIVE: | 
> http://localhost:3080/nifi-api/data-transfer/output-ports/$port-id/transactions/$tx-id/flow-files|
> |Server - SEND:   | 
> http://localhost:3080/nifi-api/data-transfer/output-ports/$port-id/transactions/$tx-id/flow-files|



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

1 2 >

1 - 100 of 135 matches

Mail list logo