[jira] [Commented] (NIFI-2027) Properties Tables

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396312#comment-15396312
 ] 

ASF subversion and git services commented on NIFI-2027:
---

Commit f3e49fefa083667fd9638dc38fa23ffa7d4d8ba0 in nifi's branch 
refs/heads/master from [~scottyaslan]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=f3e49fe ]

[NIFI-2367] Overlapping links repositioned on error/login pages
[NIFI-2025] update birdseye after dragging/dropping element on the canvas
[NIFI-2367] update width of content viewer combo
[NIFI-2355] update table sorting based on auth efforts
[NIFI-2027] update EL editors
[NIFI-2387] update bulletin alert backgroud color to actually change the color 
of the icon not the div background
[NIFI-2141] Hide bulletin icon/background on processors unless a bulletin exists
[NIFI-2400] close any open combos contained within a shell when closing the 
shell
[NIFI-2404] remove extra scrollbar from #node-events
[NIFI-2027] account for min widths of EL editors
[NIFI-2025] update birdseye after changing color
[NIFI-2027] Update EL editor checkbox text
[NIFI-2027] update checkbox text
[NIFI-2355] update table sorting
[NIFI-2141] Hide bulletin icon/background on processors unless a bulletin exists
[NIFI-2027] update EL editors
[NIFI-2367] Overlapping links repositioned on error/login pages
This closes #715


> Properties Tables
> -
>
> Key: NIFI-2027
> URL: https://issues.apache.org/jira/browse/NIFI-2027
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Scott Aslan
>Assignee: Scott Aslan
> Fix For: 1.0.0
>
> Attachments: screenshot-1.png
>
>
> - Non-EL Property editor, cannot resize horizontally
> - EL Property editor, resize renders too slowly
> - Is empty check box positioning is off
> - window resize event needs to resize any visible properties tables
> - Property table heights are too short
> - GoTo icon is right aligned
> In the property editing dialog, if it is resized very small, the layout of 
> the OK and cancel buttons get misaligned, and the OK button ends up covering 
> the resize widget. If there's no text in the box, a very small part of the 
> widget is still clickable and usable. Once there's text in the box, the user 
> is unable to click on the resize widget.
> I added a ListFile Processor. Clicked Configure, switched to Properties tab. 
> Clicked to change the value of the "Input Directory" property. When I placed 
> the mouse over the "Cancel" button, the mouse cursor changes to the 'Move' 
> cursor instead of the 'pointer' cursor



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2367) Overlapping text in Cluster pane after error

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396318#comment-15396318
 ] 

ASF GitHub Bot commented on NIFI-2367:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/715
  
Looks great @scottyaslan! This has been merged to master. Thanks!


> Overlapping text in Cluster pane after error
> 
>
> Key: NIFI-2367
> URL: https://issues.apache.org/jira/browse/NIFI-2367
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Joseph Percivall
>Assignee: Scott Aslan
> Attachments: Screen Shot 2016-07-22 at 2.42.48 PM.png
>
>
> 3 node secure cluster
> In cluster pane
> hit 500 error when refreshing pane (output in developer tools): 
> https://localhost:8481/nifi-api/controller/cluster Failed to load resource: 
> the server responded with a status of 500 (Internal Server Error)
> See overlapping text of "fix" and "home" in resulting pane (see attached 
> screenshot).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2141) Hide bulletin icon on components

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396315#comment-15396315
 ] 

ASF subversion and git services commented on NIFI-2141:
---

Commit f3e49fefa083667fd9638dc38fa23ffa7d4d8ba0 in nifi's branch 
refs/heads/master from [~scottyaslan]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=f3e49fe ]

[NIFI-2367] Overlapping links repositioned on error/login pages
[NIFI-2025] update birdseye after dragging/dropping element on the canvas
[NIFI-2367] update width of content viewer combo
[NIFI-2355] update table sorting based on auth efforts
[NIFI-2027] update EL editors
[NIFI-2387] update bulletin alert backgroud color to actually change the color 
of the icon not the div background
[NIFI-2141] Hide bulletin icon/background on processors unless a bulletin exists
[NIFI-2400] close any open combos contained within a shell when closing the 
shell
[NIFI-2404] remove extra scrollbar from #node-events
[NIFI-2027] account for min widths of EL editors
[NIFI-2025] update birdseye after changing color
[NIFI-2027] Update EL editor checkbox text
[NIFI-2027] update checkbox text
[NIFI-2355] update table sorting
[NIFI-2141] Hide bulletin icon/background on processors unless a bulletin exists
[NIFI-2027] update EL editors
[NIFI-2367] Overlapping links repositioned on error/login pages
This closes #715


> Hide bulletin icon on components 
> -
>
> Key: NIFI-2141
> URL: https://issues.apache.org/jira/browse/NIFI-2141
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Rob Moran
>Assignee: Scott Aslan
>
> When there is no bulletin for a component to call out, recommend hiding them. 
> It is consistent with current behavior. Having them visible all the time only 
> adds clutter and is a bit distracting on processors.
> When a bulletin does come in, everything should remain in place as it is now. 
> The background-color will change to the warning color (#ba554a) and the 
> bulletin icon will be white.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2355) Unable to order by any column when viewing Controller Services

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396314#comment-15396314
 ] 

ASF subversion and git services commented on NIFI-2355:
---

Commit f3e49fefa083667fd9638dc38fa23ffa7d4d8ba0 in nifi's branch 
refs/heads/master from [~scottyaslan]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=f3e49fe ]

[NIFI-2367] Overlapping links repositioned on error/login pages
[NIFI-2025] update birdseye after dragging/dropping element on the canvas
[NIFI-2367] update width of content viewer combo
[NIFI-2355] update table sorting based on auth efforts
[NIFI-2027] update EL editors
[NIFI-2387] update bulletin alert backgroud color to actually change the color 
of the icon not the div background
[NIFI-2141] Hide bulletin icon/background on processors unless a bulletin exists
[NIFI-2400] close any open combos contained within a shell when closing the 
shell
[NIFI-2404] remove extra scrollbar from #node-events
[NIFI-2027] account for min widths of EL editors
[NIFI-2025] update birdseye after changing color
[NIFI-2027] Update EL editor checkbox text
[NIFI-2027] update checkbox text
[NIFI-2355] update table sorting
[NIFI-2141] Hide bulletin icon/background on processors unless a bulletin exists
[NIFI-2027] update EL editors
[NIFI-2367] Overlapping links repositioned on error/login pages
This closes #715


> Unable to order by any column when viewing Controller Services
> --
>
> Key: NIFI-2355
> URL: https://issues.apache.org/jira/browse/NIFI-2355
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Joseph Percivall
>Assignee: Scott Aslan
>
> Have multiple controller services in a process group
> attempt to order by a column by clicking it
> nothing happens (see arrow flip)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2355) Unable to order by any column when viewing Controller Services

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396303#comment-15396303
 ] 

ASF subversion and git services commented on NIFI-2355:
---

Commit f3e49fefa083667fd9638dc38fa23ffa7d4d8ba0 in nifi's branch 
refs/heads/master from [~scottyaslan]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=f3e49fe ]

[NIFI-2367] Overlapping links repositioned on error/login pages
[NIFI-2025] update birdseye after dragging/dropping element on the canvas
[NIFI-2367] update width of content viewer combo
[NIFI-2355] update table sorting based on auth efforts
[NIFI-2027] update EL editors
[NIFI-2387] update bulletin alert backgroud color to actually change the color 
of the icon not the div background
[NIFI-2141] Hide bulletin icon/background on processors unless a bulletin exists
[NIFI-2400] close any open combos contained within a shell when closing the 
shell
[NIFI-2404] remove extra scrollbar from #node-events
[NIFI-2027] account for min widths of EL editors
[NIFI-2025] update birdseye after changing color
[NIFI-2027] Update EL editor checkbox text
[NIFI-2027] update checkbox text
[NIFI-2355] update table sorting
[NIFI-2141] Hide bulletin icon/background on processors unless a bulletin exists
[NIFI-2027] update EL editors
[NIFI-2367] Overlapping links repositioned on error/login pages
This closes #715


> Unable to order by any column when viewing Controller Services
> --
>
> Key: NIFI-2355
> URL: https://issues.apache.org/jira/browse/NIFI-2355
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Joseph Percivall
>Assignee: Scott Aslan
>
> Have multiple controller services in a process group
> attempt to order by a column by clicking it
> nothing happens (see arrow flip)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #715: [NIFI-2367] [NIFI-2025] [NIFI-2107] [NIFI-2355] [NIFI-2027]...

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/715
  
Looks great @scottyaslan! This has been merged to master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #731: NIFI-2337: Turn off logging for org.apache.curator.f...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/731

NIFI-2337: Turn off logging for org.apache.curator.framework.recipes.…

…leader.LeaderSelector since it logs only a single error, and that error 
is a but in Curator

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-2337

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/731.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #731


commit 9e9fce91eef9d3f8aa627407aa5d3677f7691c26
Author: Mark Payne 
Date:   2016-07-27T20:30:44Z

NIFI-2337: Turn off logging for 
org.apache.curator.framework.recipes.leader.LeaderSelector since it logs only a 
single error, and that error is a but in Curator




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (NIFI-2348) When processor is selected, cannot copy id from palette

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan reassigned NIFI-2348:
-

Assignee: Scott Aslan

> When processor is selected, cannot copy id from palette
> ---
>
> Key: NIFI-2348
> URL: https://issues.apache.org/jira/browse/NIFI-2348
> Project: Apache NiFi
>  Issue Type: Sub-task
>Reporter: Joseph Percivall
>Assignee: Scott Aslan
>Priority: Minor
> Fix For: 1.0.0
>
>
> Processor selected
> Processor details show up in palette
> highlight uuid in palette (processor remains "selected" on graph)
> copy then copies the processor instead of the uuid
> As a user, I would like to copy the uuid when I have the text highlighted 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72526574
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/AbstractEnrichProcessor.java
 ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public abstract class AbstractEnrichProcessor extends AbstractProcessor {
+public static final PropertyDescriptor QUERY_INPUT = new 
PropertyDescriptor.Builder()
+.name("Format the query should be to be executed ")
--- End diff --

please use name() and displayName() when defining properties [1]


https://mail-archives.apache.org/mod_mbox/nifi-dev/201605.mbox/%3c7e801f27-09cb-4c02-983d-f34ac4420...@hortonworks.com%3E


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1965) Create a QueryDNS processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396398#comment-15396398
 ] 

ASF GitHub Bot commented on NIFI-1965:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72526574
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/AbstractEnrichProcessor.java
 ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public abstract class AbstractEnrichProcessor extends AbstractProcessor {
+public static final PropertyDescriptor QUERY_INPUT = new 
PropertyDescriptor.Builder()
+.name("Format the query should be to be executed ")
--- End diff --

please use name() and displayName() when defining properties [1]


https://mail-archives.apache.org/mod_mbox/nifi-dev/201605.mbox/%3c7e801f27-09cb-4c02-983d-f34ac4420...@hortonworks.com%3E


> Create a QueryDNS processor
> ---
>
> Key: NIFI-1965
> URL: https://issues.apache.org/jira/browse/NIFI-1965
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Andre
>
> As part of a data pipeline security teams frequently must enrich data using 
> DNS enabled APIs such as:
> ShadowServer BGP and ASN lookup via DNS
> https://www.shadowserver.org/wiki/pmwiki.php/Services/IP-BGP#toc7 
> Team Cymru Malware Hash Registry
> http://www.team-cymru.org/MHR.html
> Spamhaus (SBL, XBL, etc)
> and others
> QueryDNS will use an expression language enabled property to run a query 
> against DNS and add the raw result to an attribute (for later processing if 
> necessary). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1965) Create a QueryDNS processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396447#comment-15396447
 ] 

ASF GitHub Bot commented on NIFI-1965:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72529729
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
+.description("The DNS query type to be used by the processor 
(e.g. TXT, A)")
+.required(true)
+.defaultValue("TXT")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_SERVER = new 
PropertyDescriptor.Builder()
+.name("DNS Servers")
+.description("A comma separated list of  DNS servers to be 
used. (Defaults to system wide if none is used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_TIMEOUT = new 
PropertyDescriptor.Builder()
+.name("DNS Query Timeout")
+.description("The amount of milliseconds to wait until 
considering a query as failed")
+.required(true)
+.defaultValue("1500")
+

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72529865
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
+.description("The DNS query type to be used by the processor 
(e.g. TXT, A)")
+.required(true)
+.defaultValue("TXT")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_SERVER = new 
PropertyDescriptor.Builder()
+.name("DNS Servers")
+.description("A comma separated list of  DNS servers to be 
used. (Defaults to system wide if none is used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_TIMEOUT = new 
PropertyDescriptor.Builder()
+.name("DNS Query Timeout")
+.description("The amount of milliseconds to wait until 
considering a query as failed")
+.required(true)
+.defaultValue("1500")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_RETRIES = new 
PropertyDescriptor.Builder()
+.name("DNS Query Retries")
+

[jira] [Commented] (NIFI-1965) Create a QueryDNS processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396461#comment-15396461
 ] 

ASF GitHub Bot commented on NIFI-1965:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72530345
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/test/java/org/apache/nifi/processors/enrich/TestQueryDNS.java
 ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+
+import javax.naming.Context;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttribute;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+
+import static org.junit.Assert.assertTrue;
+
+public class TestQueryDNS  {
+private QueryDNS queryDNS;
+private TestRunner queryDNSTestRunner;
+
+@Before
+public void setupTest() throws Exception {
+this.queryDNS =  new QueryDNS();
+this.queryDNSTestRunner = TestRunners.newTestRunner(queryDNS);
+
+Hashtable env = new Hashtable();
+env.put(Context.INITIAL_CONTEXT_FACTORY, 
FakeDNSInitialDirContextFactory.class.getName());
+
+this.queryDNS.initializeContext(env);
+
+final DirContext mockContext = 
FakeDNSInitialDirContextFactory.getLatestMockContext();
+
+// Capture JNDI's getAttibutes method containing the (String) 
queryValue and (String[]) queryType
+Mockito.when( mockContext.getAttributes(Mockito.anyString(), 
Mockito.any(String[].class)))
+.thenAnswer(new Answer() {
+public Object answer(InvocationOnMock invocation) 
throws Throwable {
+// Craft a false DNS response
+// Note the DNS response will not make use of any 
of the mocked
+// query contents (all input is discarded and 
replies synthetically
+// generated
+return craftResponse(invocation);
+}
+});
+}
+
+@Test
+public void testVanillaQueryWithoutSplit()  {
+queryDNSTestRunner.setProperty(QueryDNS.DNS_QUERY_TYPE, "PTR");
+queryDNSTestRunner.setProperty(QueryDNS.DNS_RETRIES, "1");
+queryDNSTestRunner.setProperty(QueryDNS.DNS_TIMEOUT, "1000");
+queryDNSTestRunner.setProperty(QueryDNS.QUERY_INPUT, 
"${ip_address:getDelimitedField(4, '.'):trim()}" +
+".${ip_address:getDelimitedField(3, '.'):trim()}" +
+".${ip_address:getDelimitedField(2, '.'):trim()}" +
+".${ip_address:getDelimitedField(1, '.'):trim()}" +
+".in-addr.arpa");
+queryDNSTestRunner.setProperty(QueryDNS.QUERY_PARSER, 
QueryDNS.NONE.getValue());
+
+final Map attributeMap = new HashMap<>();
+attributeMap.put("ip_address", "123.123.123.123");
+
+queryDNSTestRunner.enqueue(new byte[0], attributeMap);
+queryDNSTestRunner.enqueue("teste teste teste 
chocolate".getBytes());
+
+queryDNSTestRunner.run(1,true, false);
+
+List results = 
queryDNSTestRunner.getFlowFilesForRelationship(QueryDNS.REL_FOUND);
+assertTrue(results.size() == 1);
+String result = 

[jira] [Updated] (NIFI-2224) Upload template dialog does not show template name selected

[Joseph Percivall (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall updated NIFI-2224:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Upload template dialog does not show template name selected
> ---
>
> Key: NIFI-2224
> URL: https://issues.apache.org/jira/browse/NIFI-2224
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Affects Versions: 1.0.0
> Environment: Mac OS X; Chrome
>Reporter: Andrew Lim
>Assignee: Matt Gilman
>  Labels: UI
> Fix For: 1.0.0
>
> Attachments: NIFI-2224.mp4
>
>
> Following STRs are shown in the attached video:
> 1. Select Upload Template from operator palette
> 2. Select magnifying glass icon to browse for template
> 3. Select template '1.xml'
> 4. Select open button
> 5. Upload template dialog in NiFi shows '1.xml' as selected
> 6. Select Upload button
> 7. Template uploaded successfully
> 8. Select Upload Template from operator palette
> 9. Select magnifying glass icon to browse for template
> 10 Select template '1.xml' AGAIN
> 11. Select open button
> 12. Upload template dialog in NiFi does NOT show '1.xml' as selected
> Separate bug (NIFI-2223)
> 13.  Select Upload button
> 14. Template uploaded successfully



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2175) Uploading a template (via Operator palette) has improper error checking

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396467#comment-15396467
 ] 

ASF subversion and git services commented on NIFI-2175:
---

Commit ae344806c0cdbe81104c339d9e810fb3cff1a3c1 in nifi's branch 
refs/heads/master from [~mcgilman]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ae34480 ]

NIFI-2224: - Ensuring the template form is reset when the upload template 
dialog is closed. NIFI-2175: - Not submitting the template form is a template 
isn't choosen. NIFI-2176: - Ensuring a template is specified during creation. 
NIFI-2223: - Ensuring templates with the same name cannot be added. NIFI-2296: 
- Updating the tooltip for the upload template browse button.

- Cleaning up un-used parameters to REST endpoints.

This closes #725

Signed-off-by: jpercivall 


> Uploading a template (via Operator palette) has improper error checking
> ---
>
> Key: NIFI-2175
> URL: https://issues.apache.org/jira/browse/NIFI-2175
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Affects Versions: 1.0.0
> Environment: Mac OS X, Chrome
>Reporter: Andrew Lim
>Assignee: Matt Gilman
>  Labels: UI
> Fix For: 1.0.0
>
> Attachments: NIFI-2175_notemplateselected.png
>
>
> Issue 1:
> Select the "Upload Template" button in the Operator palette.  Immediately 
> select "Upload".  The error "The specified template is not in a valid 
> format." is shown even though the user has not selected a template.  See 
> attached screenshot.
> Issue 2:
> This time, select a valid template to upload and select "Upload".  The 
> Success message "Template successfully imported." is displayed.  Select "OK".
> Now select "Upload Template" button from the Operator palette again.  
> Immediately select "Upload".  The Success message "Template successfully 
> imported." is displayed even though no template was selected.
> If you go to the Templates window via the management drop-down, you can see 
> that the template selected earlier was imported again even though it wasn't 
> explicitly chosen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2367) Overlapping text in Cluster pane after error

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan updated NIFI-2367:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Overlapping text in Cluster pane after error
> 
>
> Key: NIFI-2367
> URL: https://issues.apache.org/jira/browse/NIFI-2367
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Joseph Percivall
>Assignee: Scott Aslan
> Attachments: Screen Shot 2016-07-22 at 2.42.48 PM.png
>
>
> 3 node secure cluster
> In cluster pane
> hit 500 error when refreshing pane (output in developer tools): 
> https://localhost:8481/nifi-api/controller/cluster Failed to load resource: 
> the server responded with a status of 500 (Internal Server Error)
> See overlapping text of "fix" and "home" in resulting pane (see attached 
> screenshot).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #732: [NIFI-2396] Update Bulletin Board shell header and f...

[scottyaslan]

GitHub user scottyaslan opened a pull request:

https://github.com/apache/nifi/pull/732

[NIFI-2396] Update Bulletin Board shell header and filter control pos…

…ition

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scottyaslan/nifi responsiveDevBranch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/732.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #732


commit b0c10b0c6284bac8a03fd52b3030b679ff19fc85
Author: Scott Aslan 
Date:   2016-07-27T20:41:18Z

[NIFI-2396] Update Bulletin Board shell header and filter control position




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72526705
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/AbstractEnrichProcessor.java
 ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public abstract class AbstractEnrichProcessor extends AbstractProcessor {
+public static final PropertyDescriptor QUERY_INPUT = new 
PropertyDescriptor.Builder()
+.name("Format the query should be to be executed ")
+.required(true)
+.description("The value that should be used to populate the 
query")
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final AllowableValue SPLIT= new AllowableValue("Split", 
"Split",
+"String.splitUse a delimiter character or RegEx  to split the 
results into attributes");
--- End diff --

wording?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1965) Create a QueryDNS processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396454#comment-15396454
 ] 

ASF GitHub Bot commented on NIFI-1965:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72530181
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
+.description("The DNS query type to be used by the processor 
(e.g. TXT, A)")
+.required(true)
+.defaultValue("TXT")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_SERVER = new 
PropertyDescriptor.Builder()
+.name("DNS Servers")
+.description("A comma separated list of  DNS servers to be 
used. (Defaults to system wide if none is used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_TIMEOUT = new 
PropertyDescriptor.Builder()
+.name("DNS Query Timeout")
+.description("The amount of milliseconds to wait until 
considering a query as failed")
+.required(true)
+.defaultValue("1500")
+

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72530345
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/test/java/org/apache/nifi/processors/enrich/TestQueryDNS.java
 ---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+
+import javax.naming.Context;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttribute;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+
+import static org.junit.Assert.assertTrue;
+
+public class TestQueryDNS  {
+private QueryDNS queryDNS;
+private TestRunner queryDNSTestRunner;
+
+@Before
+public void setupTest() throws Exception {
+this.queryDNS =  new QueryDNS();
+this.queryDNSTestRunner = TestRunners.newTestRunner(queryDNS);
+
+Hashtable env = new Hashtable();
+env.put(Context.INITIAL_CONTEXT_FACTORY, 
FakeDNSInitialDirContextFactory.class.getName());
+
+this.queryDNS.initializeContext(env);
+
+final DirContext mockContext = 
FakeDNSInitialDirContextFactory.getLatestMockContext();
+
+// Capture JNDI's getAttibutes method containing the (String) 
queryValue and (String[]) queryType
+Mockito.when( mockContext.getAttributes(Mockito.anyString(), 
Mockito.any(String[].class)))
+.thenAnswer(new Answer() {
+public Object answer(InvocationOnMock invocation) 
throws Throwable {
+// Craft a false DNS response
+// Note the DNS response will not make use of any 
of the mocked
+// query contents (all input is discarded and 
replies synthetically
+// generated
+return craftResponse(invocation);
+}
+});
+}
+
+@Test
+public void testVanillaQueryWithoutSplit()  {
+queryDNSTestRunner.setProperty(QueryDNS.DNS_QUERY_TYPE, "PTR");
+queryDNSTestRunner.setProperty(QueryDNS.DNS_RETRIES, "1");
+queryDNSTestRunner.setProperty(QueryDNS.DNS_TIMEOUT, "1000");
+queryDNSTestRunner.setProperty(QueryDNS.QUERY_INPUT, 
"${ip_address:getDelimitedField(4, '.'):trim()}" +
+".${ip_address:getDelimitedField(3, '.'):trim()}" +
+".${ip_address:getDelimitedField(2, '.'):trim()}" +
+".${ip_address:getDelimitedField(1, '.'):trim()}" +
+".in-addr.arpa");
+queryDNSTestRunner.setProperty(QueryDNS.QUERY_PARSER, 
QueryDNS.NONE.getValue());
+
+final Map attributeMap = new HashMap<>();
+attributeMap.put("ip_address", "123.123.123.123");
+
+queryDNSTestRunner.enqueue(new byte[0], attributeMap);
+queryDNSTestRunner.enqueue("teste teste teste 
chocolate".getBytes());
+
+queryDNSTestRunner.run(1,true, false);
+
+List results = 
queryDNSTestRunner.getFlowFilesForRelationship(QueryDNS.REL_FOUND);
+assertTrue(results.size() == 1);
+String result = 
results.get(0).getAttribute("enrich.dns.record0.group0");
+
+assertTrue(result.contains("apache.nifi.org"));
+
+
+}
+
+@Test
+public void testValidDataWithSplit()  {
+

[jira] [Commented] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395721#comment-15395721
 ] 

Mark Payne commented on NIFI-2395:
--

Brian - I have looked into this. I believe the condition in the while loop is 
actually correct. If we get rid of the Java syntax and focus on what it means 
in English, it is saying:

while ("there are events on the queue" OR "more events will be added to the 
queue") {
   // wait for an event and index it
}

So I do not believe the case is that the thread evaluated the logic and 
finished before any messages were added to the queue.

Rather, my suspicion is that the call the indexingAction.index() threw an 
Exception. Unfortunately, if that occurs, the result is that the task would 
terminate prematurely, and this could result in
the other thread blocking on queue.offer().

I think we can address pretty easily by updating the code that calls 
queue.offer() such that if queue.offer() returns false we check the 'futures' 
list to see if any futures have terminated Exceptionally. And if so, the
thread that calls queue.offer() should re-throw that exception. This is exactly 
what happens after this thread has finished calling queue.offer() but as you 
are seeing, I think we need to also do it if we are unable to
add events to the queue.

I will work on getting a PR in that addresses this issue. Do you agree that 
this makes sense?


> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Blocker
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} bytes)", 
> journalFileCount, repoSize, journalCountThreshold, sizeThreshold);
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> }
> logger.info("Provenance Repository has now caught up with 
> rolling over journal files. Current number of "
> + "journal files to be rolled over is {}", 
> journalFileCount);
> }
> {code}
> My nifi is at the sleep indefinitely.  The reason my nifi cannot move forward 
> is because of the thread doing the merge is stopped.  The thread doing the 
> merge is at:
> {code}
> accepted = eventQueue.offer(new Tuple<>(record, blockIndex), 10, 
> TimeUnit.MILLISECONDS);
> {code}
> so the queue is full.  
> What I believe happened is that the callables created here:
> 

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72462023
  
--- Diff: nifi-toolkit/nifi-toolkit-assembly/NOTICE ---
@@ -0,0 +1,112 @@
+Apache NiFi Toolkit
+Copyright 2014-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===
+Apache Software License v2
+===
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Apache NiFi
+The following NOTICE information applies:
+  Apache NiFi
+  Copyright 2014-2016 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+  This product includes the following work from the Apache Hadoop 
project:
+
+  BoundedByteArrayOutputStream.java adapted to 
SoftLimitBoundedByteArrayOutputStream.java
+
+  (ASLv2) Apache Commons CLI
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2001-2015 The Apache Software Foundation
+
+  (ASLv2) Apache Commons Codec
+The following NOTICE information applies:
+  Apache Commons Codec
+  Copyright 2002-2014 The Apache Software Foundation
+
+  src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java
+  contains test data from http://aspell.net/test/orig/batch0.tab.
+  Copyright (C) 2002 Kevin Atkinson (kev...@gnu.org)
+
+  
===
+
+  The content of package org.apache.commons.codec.language.bm has been 
translated
+  from the original php source code available at 
http://stevemorse.org/phoneticinfo.htm
+  with permission from the original authors.
+  Original source copyright:
+  Copyright (c) 2008 Alexander Beider & Stephen P. Morse.
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2012 The Apache Software Foundation
+
+  (ASLv2) Apache Commons Lang
--- End diff --

I don't see where this is brought in as a dependency.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395893#comment-15395893
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user brosander commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72468676
  
--- Diff: nifi-toolkit/nifi-toolkit-assembly/pom.xml ---
@@ -0,0 +1,89 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;>
+4.0.0
+
+org.apache.nifi
+nifi-toolkit
+1.0.0-SNAPSHOT
+
+nifi-toolkit-assembly
+pom
+This is the assembly Apache NiFi Toolkit
+
+
+
+org.apache.rat
+apache-rat-plugin
+
+
+
src/main/resources/conf/config-client.json
+
src/main/resources/conf/config-server.json
+
+
+
+
+maven-assembly-plugin
+
+nifi-toolkit-${project.version}
+
+
+
+make shared resource
+
+single
+
+package
+
+
+
0755
+0755
+0644
+
+
+
src/main/assembly/dependencies.xml
+
+posix
+
+
+
+
+
+
+
+
+org.apache.nifi
+nifi-toolkit-tls
+
+
+org.slf4j
+slf4j-api
+compile
+1.7.12
+
+
+org.eclipse.jetty
+jetty-server
+compile
+
+
+javax.servlet
+javax.servlet-api
+compile
+
+
+commons-io
+commons-io
--- End diff --

It was being marked provided before, I can dig in more


> Command Line Keystore and Truststore utility
> 
>
> Key: NIFI-2193
> URL: https://issues.apache.org/jira/browse/NIFI-2193
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Bryan Rosander
>Assignee: Bryan Rosander
>
> In order to facilitate secure setup of NiFi, it would be useful to have a 
> command line utility capable of generating the required keystores, 
> truststore, and relevant configuration files.
> It should be able to generate keystores for each NiFi node, a truststore that 
> they all use, and relevant passwords and configuration files for using the 
> keystores and truststore.
> Additionally, in order to support distributed deployment, a web based 
> certificate authority with corresponding client will allow for each NiFi 
> instance to generate its own keypair and then request signing by the CA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2347) Pasted Components should inherit policies from the original, not the group it's pasted into

[Matt Gilman (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395904#comment-15395904
 ] 

Matt Gilman commented on NIFI-2347:
---

In addition to the policies of the component, we also need to include policies 
for the data of the component, data transfers of the component (root group 
ports - site to site), and policies of the component.

> Pasted Components should inherit policies from the original, not the group 
> it's pasted into
> ---
>
> Key: NIFI-2347
> URL: https://issues.apache.org/jira/browse/NIFI-2347
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Percivall
>Assignee: Matt Gilman
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Copy a processor that you have view permissions but not write
> Paste it into a group that you have full permissions
> Pasted processor inherits from the group and gives you full permissions.
> Instead it should copy/inherit the policies from the copied component.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395905#comment-15395905
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user brosander commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72469932
  
--- Diff: nifi-toolkit/nifi-toolkit-assembly/NOTICE ---
@@ -0,0 +1,112 @@
+Apache NiFi Toolkit
+Copyright 2014-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===
+Apache Software License v2
+===
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Apache NiFi
+The following NOTICE information applies:
+  Apache NiFi
+  Copyright 2014-2016 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+  This product includes the following work from the Apache Hadoop 
project:
+
+  BoundedByteArrayOutputStream.java adapted to 
SoftLimitBoundedByteArrayOutputStream.java
+
+  (ASLv2) Apache Commons CLI
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2001-2015 The Apache Software Foundation
+
+  (ASLv2) Apache Commons Codec
+The following NOTICE information applies:
+  Apache Commons Codec
+  Copyright 2002-2014 The Apache Software Foundation
+
+  src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java
+  contains test data from http://aspell.net/test/orig/batch0.tab.
+  Copyright (C) 2002 Kevin Atkinson (kev...@gnu.org)
+
+  
===
+
+  The content of package org.apache.commons.codec.language.bm has been 
translated
+  from the original php source code available at 
http://stevemorse.org/phoneticinfo.htm
+  with permission from the original authors.
+  Original source copyright:
+  Copyright (c) 2008 Alexander Beider & Stephen P. Morse.
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2012 The Apache Software Foundation
+
+  (ASLv2) Apache Commons Lang
+The following NOTICE information applies:
+  Apache Commons Lang
+  Copyright 2001-2015 The Apache Software Foundation
+
+  (ASLv2) Apache Commons Logging
+The following NOTICE information applies:
+  Apache Commons Logging
+  Copyright 2003-2014 The Apache Software Foundation
+
+  (ASLv2) Apache HttpComponents
+The following NOTICE information applies:
+  Apache HttpClient
+  Copyright 1999-2015 The Apache Software Foundation
+
+  Apache HttpCore
+  Copyright 2005-2015 The Apache Software Foundation
+
+  This project contains annotations derived from JCIP-ANNOTATIONS
+  Copyright (c) 2005 Brian Goetz and Tim Peierls. See 
http://www.jcip.net
+
+  (ASLv2) Jackson JSON processor
+The following NOTICE information applies:
+  # Jackson JSON processor
+
+  Jackson is a high-performance, Free/Open Source JSON processing 
library.
+  It was originally written by Tatu Saloranta (tatu.salora...@iki.fi), 
and has
+  been in development since 2007.
+  It is currently developed by a community of developers, as well as 
supported
+  commercially by FasterXML.com.
+
+  ## Licensing
+
+  Jackson core and extension components may licensed under different 
licenses.
+  To find the details that apply to this artifact see the accompanying 
LICENSE file.
+  For more information, including possible other licensing options, 
contact
+  FasterXML.com (http://fasterxml.com).
+
+  ## Credits
+
+  A list of contributors may be found from CREDITS file, which is 
included
+  in some artifacts (usually source distributions); but is always 
available
+  from the source code management (SCM) system project uses.
+
+  (ASLv2) Jetty
+The following NOTICE information applies:
+   Jetty Web Container
+   Copyright 1995-2015 Mort Bay Consulting Pty Ltd.
+
+  (ASLv2) Apache log4j
+The following NOTICE information applies:
+  Apache log4j
+  Copyright 2007 The Apache Software Foundation
+
+===
+MIT License
+===
+
 

[GitHub] nifi issue #695: NIFI-2193 - Command line SSL config utility as well as cert...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/695
  
I don't really understand the structure of the tool here -- it looks like 
`TlsToolkitMain` is invoked from the scripts, and it has an internal instance 
of `TlsToolkitCommandLine` to parse the command line arguments, and an instance 
of `TlsHelper`. I would expect `TlsToolkitMain` to be a standalone class which 
encapsulated the logic (and ideally, to be named something like 
`CertificateGeneratorTool` as it is focusing solely on the key and certificate 
generation), and `TlsToolkitCommandLine` to be a wrapper class which handles 
and parses command line input to then invoke the main class logic. This way, 
the logic could be invoked programmatically from other classes with direct 
parameter provision. It seems to me that the current structure tightly couples 
all certificate generation & keystore population the with command line entry 
point. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396726#comment-15396726
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72548970
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/util/TlsHelper.java
 ---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.util;
+
+import org.apache.nifi.security.util.CertificateUtils;
+import org.apache.nifi.toolkit.tls.commandLine.TlsToolkitCommandLine;
+import org.apache.nifi.toolkit.tls.configuration.TlsHelperConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.cert.crmf.CRMFException;
+import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter;
+import org.bouncycastle.cert.jcajce.JcaX509ExtensionUtils;
+import org.bouncycastle.eac.EACException;
+import org.bouncycastle.jce.provider.BouncyCastleProvider;
+import org.bouncycastle.openssl.PEMParser;
+import org.bouncycastle.openssl.jcajce.JcaMiscPEMGenerator;
+import org.bouncycastle.operator.OperatorCreationException;
+import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequestBuilder;
+import org.bouncycastle.util.io.pem.PemWriter;
+
+import javax.crypto.Mac;
+import javax.crypto.spec.SecretKeySpec;
+import javax.security.auth.x500.X500Principal;
+import java.io.IOException;
+import java.io.StringReader;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.security.GeneralSecurityException;
+import java.security.InvalidKeyException;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.PublicKey;
+import java.security.Security;
+import java.security.SignatureException;
+import java.security.cert.Certificate;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.security.spec.InvalidKeySpecException;
+
+public class TlsHelper {
+public static final String PROVIDER = 
BouncyCastleProvider.PROVIDER_NAME;
+public static final String PKCS12 = "PKCS12";
+private final KeyPairGenerator keyPairGenerator;
+private final int days;
+private final String signingAlgorithm;
+
+public TlsHelper(TlsHelperConfig tlsHelperConfig) throws 
NoSuchAlgorithmException {
+this(tlsHelperConfig.getDays(), tlsHelperConfig.getKeySize(), 
tlsHelperConfig.getKeyPairAlgorithm(), tlsHelperConfig.getSigningAlgorithm());
+}
+
+public TlsHelper(TlsToolkitCommandLine tlsToolkitCommandLine) throws 
NoSuchAlgorithmException {
+this(tlsToolkitCommandLine.getTlsHelperConfig());
+}
+
+public TlsHelper(int days, int keySize, String keyPairAlgorithm, 
String signingAlgorithm) throws NoSuchAlgorithmException {
+this(createKeyPairGenerator(keyPairAlgorithm, keySize), days, 
signingAlgorithm);
+}
+
+protected TlsHelper(KeyPairGenerator keyPairGenerator, int days, 
String signingAlgorithm) {
+this.keyPairGenerator = keyPairGenerator;
+this.days = days;
+this.signingAlgorithm = signingAlgorithm;
+}
+
+public static void addBouncyCastleProvider() {
+Security.addProvider(new BouncyCastleProvider());
+}
+
+private static KeyPairGenerator 

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72548970
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/util/TlsHelper.java
 ---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.util;
+
+import org.apache.nifi.security.util.CertificateUtils;
+import org.apache.nifi.toolkit.tls.commandLine.TlsToolkitCommandLine;
+import org.apache.nifi.toolkit.tls.configuration.TlsHelperConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.cert.crmf.CRMFException;
+import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter;
+import org.bouncycastle.cert.jcajce.JcaX509ExtensionUtils;
+import org.bouncycastle.eac.EACException;
+import org.bouncycastle.jce.provider.BouncyCastleProvider;
+import org.bouncycastle.openssl.PEMParser;
+import org.bouncycastle.openssl.jcajce.JcaMiscPEMGenerator;
+import org.bouncycastle.operator.OperatorCreationException;
+import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequestBuilder;
+import org.bouncycastle.util.io.pem.PemWriter;
+
+import javax.crypto.Mac;
+import javax.crypto.spec.SecretKeySpec;
+import javax.security.auth.x500.X500Principal;
+import java.io.IOException;
+import java.io.StringReader;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.security.GeneralSecurityException;
+import java.security.InvalidKeyException;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.PublicKey;
+import java.security.Security;
+import java.security.SignatureException;
+import java.security.cert.Certificate;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.security.spec.InvalidKeySpecException;
+
+public class TlsHelper {
+public static final String PROVIDER = 
BouncyCastleProvider.PROVIDER_NAME;
+public static final String PKCS12 = "PKCS12";
+private final KeyPairGenerator keyPairGenerator;
+private final int days;
+private final String signingAlgorithm;
+
+public TlsHelper(TlsHelperConfig tlsHelperConfig) throws 
NoSuchAlgorithmException {
+this(tlsHelperConfig.getDays(), tlsHelperConfig.getKeySize(), 
tlsHelperConfig.getKeyPairAlgorithm(), tlsHelperConfig.getSigningAlgorithm());
+}
+
+public TlsHelper(TlsToolkitCommandLine tlsToolkitCommandLine) throws 
NoSuchAlgorithmException {
+this(tlsToolkitCommandLine.getTlsHelperConfig());
+}
+
+public TlsHelper(int days, int keySize, String keyPairAlgorithm, 
String signingAlgorithm) throws NoSuchAlgorithmException {
+this(createKeyPairGenerator(keyPairAlgorithm, keySize), days, 
signingAlgorithm);
+}
+
+protected TlsHelper(KeyPairGenerator keyPairGenerator, int days, 
String signingAlgorithm) {
+this.keyPairGenerator = keyPairGenerator;
+this.days = days;
+this.signingAlgorithm = signingAlgorithm;
+}
+
+public static void addBouncyCastleProvider() {
+Security.addProvider(new BouncyCastleProvider());
+}
+
+private static KeyPairGenerator createKeyPairGenerator(String 
algorithm, int keySize) throws NoSuchAlgorithmException {
+KeyPairGenerator instance = 
KeyPairGenerator.getInstance(algorithm);
+instance.initialize(keySize);
+return instance;
+}
  

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72549313
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/util/TlsHelper.java
 ---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.util;
+
+import org.apache.nifi.security.util.CertificateUtils;
+import org.apache.nifi.toolkit.tls.commandLine.TlsToolkitCommandLine;
+import org.apache.nifi.toolkit.tls.configuration.TlsHelperConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.cert.crmf.CRMFException;
+import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter;
+import org.bouncycastle.cert.jcajce.JcaX509ExtensionUtils;
+import org.bouncycastle.eac.EACException;
+import org.bouncycastle.jce.provider.BouncyCastleProvider;
+import org.bouncycastle.openssl.PEMParser;
+import org.bouncycastle.openssl.jcajce.JcaMiscPEMGenerator;
+import org.bouncycastle.operator.OperatorCreationException;
+import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequestBuilder;
+import org.bouncycastle.util.io.pem.PemWriter;
+
+import javax.crypto.Mac;
+import javax.crypto.spec.SecretKeySpec;
+import javax.security.auth.x500.X500Principal;
+import java.io.IOException;
+import java.io.StringReader;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.security.GeneralSecurityException;
+import java.security.InvalidKeyException;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.PublicKey;
+import java.security.Security;
+import java.security.SignatureException;
+import java.security.cert.Certificate;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.security.spec.InvalidKeySpecException;
+
+public class TlsHelper {
+public static final String PROVIDER = 
BouncyCastleProvider.PROVIDER_NAME;
+public static final String PKCS12 = "PKCS12";
+private final KeyPairGenerator keyPairGenerator;
+private final int days;
+private final String signingAlgorithm;
+
+public TlsHelper(TlsHelperConfig tlsHelperConfig) throws 
NoSuchAlgorithmException {
+this(tlsHelperConfig.getDays(), tlsHelperConfig.getKeySize(), 
tlsHelperConfig.getKeyPairAlgorithm(), tlsHelperConfig.getSigningAlgorithm());
+}
+
+public TlsHelper(TlsToolkitCommandLine tlsToolkitCommandLine) throws 
NoSuchAlgorithmException {
+this(tlsToolkitCommandLine.getTlsHelperConfig());
+}
+
+public TlsHelper(int days, int keySize, String keyPairAlgorithm, 
String signingAlgorithm) throws NoSuchAlgorithmException {
+this(createKeyPairGenerator(keyPairAlgorithm, keySize), days, 
signingAlgorithm);
+}
+
+protected TlsHelper(KeyPairGenerator keyPairGenerator, int days, 
String signingAlgorithm) {
+this.keyPairGenerator = keyPairGenerator;
+this.days = days;
+this.signingAlgorithm = signingAlgorithm;
+}
+
+public static void addBouncyCastleProvider() {
+Security.addProvider(new BouncyCastleProvider());
+}
+
+private static KeyPairGenerator createKeyPairGenerator(String 
algorithm, int keySize) throws NoSuchAlgorithmException {
+KeyPairGenerator instance = 
KeyPairGenerator.getInstance(algorithm);
+instance.initialize(keySize);
+return instance;
+}
  

[GitHub] nifi pull request #715: [NIFI-2367] [NIFI-2025] [NIFI-2107] [NIFI-2355] [NIF...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/715


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2025) Birdseye does not update when repositioning an existing item on the canvas

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan updated NIFI-2025:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Birdseye does not update when repositioning an existing item on the canvas
> --
>
> Key: NIFI-2025
> URL: https://issues.apache.org/jira/browse/NIFI-2025
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Scott Aslan
>Assignee: Scott Aslan
> Fix For: 1.0.0
>
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> Birdseye does not update when repositioning an existing item on the canvas 
> but it does update appropriately when adding/deleting items from the canvas. 
> We need to refresh the birdseye when repositioning items on the canvas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2141) Hide bulletin icon on components

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan updated NIFI-2141:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Hide bulletin icon on components 
> -
>
> Key: NIFI-2141
> URL: https://issues.apache.org/jira/browse/NIFI-2141
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Rob Moran
>Assignee: Scott Aslan
>
> When there is no bulletin for a component to call out, recommend hiding them. 
> It is consistent with current behavior. Having them visible all the time only 
> adds clutter and is a bit distracting on processors.
> When a bulletin does come in, everything should remain in place as it is now. 
> The background-color will change to the warning color (#ba554a) and the 
> bulletin icon will be white.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2387) Controller Service bulletin highlighting is "off"

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan updated NIFI-2387:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Controller Service bulletin highlighting is "off"
> -
>
> Key: NIFI-2387
> URL: https://issues.apache.org/jira/browse/NIFI-2387
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Joseph Percivall
>Assignee: Scott Aslan
> Attachments: Screen Shot 2016-07-25 at 10.22.17 AM.png
>
>
> View root level controller services
> start a DistributedMapCacheServer with the same port as the NiFi instance
> See a red square partially covering the "bulletin" icon
> See attached image



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72530250
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
+.description("The DNS query type to be used by the processor 
(e.g. TXT, A)")
+.required(true)
+.defaultValue("TXT")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_SERVER = new 
PropertyDescriptor.Builder()
+.name("DNS Servers")
+.description("A comma separated list of  DNS servers to be 
used. (Defaults to system wide if none is used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_TIMEOUT = new 
PropertyDescriptor.Builder()
+.name("DNS Query Timeout")
+.description("The amount of milliseconds to wait until 
considering a query as failed")
+.required(true)
+.defaultValue("1500")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_RETRIES = new 
PropertyDescriptor.Builder()
+.name("DNS Query Retries")
+

[GitHub] nifi pull request #496: NIFI-1965 - Implement QueryDNS Processor

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/496#discussion_r72530181
  
--- Diff: 
nifi-nar-bundles/nifi-enrich-bundle/nifi-enrich-processors/src/main/java/org/apache/nifi/processors/enrich/QueryDNS.java
 ---
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.enrich;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.naming.Context;
+import javax.naming.NameNotFoundException;
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import javax.naming.directory.Attributes;
+import javax.naming.directory.BasicAttributes;
+import javax.naming.directory.DirContext;
+import javax.naming.directory.InitialDirContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"dns", "enrich", "ip"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("A powerful DNS query processor primary designed to 
enrich DataFlows with DNS based APIs " +
+"(e.g. RBLs, ShadowServer's ASN lookup) but that can be also used 
to perform regular DNS lookups.")
+@WritesAttributes({
+@WritesAttribute(attribute = "enrich.dns.record*.group*", 
description = "The captured fields of the DNS query response for each of the 
records received"),
+})
+public class QueryDNS extends AbstractEnrichProcessor {
+
+public static final PropertyDescriptor DNS_QUERY_TYPE = new 
PropertyDescriptor.Builder()
+.name("DNS Query Type")
+.description("The DNS query type to be used by the processor 
(e.g. TXT, A)")
+.required(true)
+.defaultValue("TXT")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_SERVER = new 
PropertyDescriptor.Builder()
+.name("DNS Servers")
+.description("A comma separated list of  DNS servers to be 
used. (Defaults to system wide if none is used)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_TIMEOUT = new 
PropertyDescriptor.Builder()
+.name("DNS Query Timeout")
+.description("The amount of milliseconds to wait until 
considering a query as failed")
+.required(true)
+.defaultValue("1500")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DNS_RETRIES = new 
PropertyDescriptor.Builder()
+.name("DNS Query Retries")
+

[jira] [Updated] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2395:
--
Assignee: Mark Payne  (was: Joseph Witt)

> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0, 0.8.0
>
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} bytes)", 
> journalFileCount, repoSize, journalCountThreshold, sizeThreshold);
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> }
> logger.info("Provenance Repository has now caught up with 
> rolling over journal files. Current number of "
> + "journal files to be rolled over is {}", 
> journalFileCount);
> }
> {code}
> My nifi is at the sleep indefinitely.  The reason my nifi cannot move forward 
> is because of the thread doing the merge is stopped.  The thread doing the 
> merge is at:
> {code}
> accepted = eventQueue.offer(new Tuple<>(record, blockIndex), 10, 
> TimeUnit.MILLISECONDS);
> {code}
> so the queue is full.  
> What I believe happened is that the callables created here:
> {code}
> final Callable callable = new 
> Callable() {
> @Override
> public Object call() throws IOException {
> while (!eventQueue.isEmpty() || 
> !finishedAdding.get()) {
> final 
> Tuple tuple;
> try {
> tuple = eventQueue.poll(10, 
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException 
> ie) {
> continue;
> }
> if (tuple == null) {
> continue;
> }
> indexingAction.index(tuple.getKey(), 
> indexWriter, tuple.getValue());
> }
> return null;
> }
> {code}
> finish before the offer adds its first event because I 

[jira] [Updated] (NIFI-2328) Update UI for JoltTransform processor

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2328:
--
Fix Version/s: (was: 1.0.0)

> Update UI for JoltTransform processor
> -
>
> Key: NIFI-2328
> URL: https://issues.apache.org/jira/browse/NIFI-2328
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>
> UI of the JoltTransform processor is un-finished as-is, need to update it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2328) Update UI for JoltTransform processor

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396524#comment-15396524
 ] 

Joseph Witt commented on NIFI-2328:
---

removed fix version for this until someone is able to make progress on it.

> Update UI for JoltTransform processor
> -
>
> Key: NIFI-2328
> URL: https://issues.apache.org/jira/browse/NIFI-2328
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>
> UI of the JoltTransform processor is un-finished as-is, need to update it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396558#comment-15396558
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/695
  
I am reviewing, focusing on the cryptographic code in here as well as the 
token signing for communication between the nodes and CA over an untrusted 
channel. 

@brosander can you please provide some documentation/admin guide? The only 
thing I have seen so far is the usage in the NiFiToolkitTLS class. 


> Command Line Keystore and Truststore utility
> 
>
> Key: NIFI-2193
> URL: https://issues.apache.org/jira/browse/NIFI-2193
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Bryan Rosander
>Assignee: Bryan Rosander
>
> In order to facilitate secure setup of NiFi, it would be useful to have a 
> command line utility capable of generating the required keystores, 
> truststore, and relevant configuration files.
> It should be able to generate keystores for each NiFi node, a truststore that 
> they all use, and relevant passwords and configuration files for using the 
> keystores and truststore.
> Additionally, in order to support distributed deployment, a web based 
> certificate authority with corresponding client will allow for each NiFi 
> instance to generate its own keypair and then request signing by the CA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #476: NIFI-1942 Processor to validate CSV against user-sup...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/476#discussion_r72539247
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateCsv.java
 ---
@@ -0,0 +1,613 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.Reader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.supercsv.cellprocessor.Optional;
+import org.supercsv.cellprocessor.ParseBigDecimal;
+import org.supercsv.cellprocessor.ParseBool;
+import org.supercsv.cellprocessor.ParseChar;
+import org.supercsv.cellprocessor.ParseDate;
+import org.supercsv.cellprocessor.ParseDouble;
+import org.supercsv.cellprocessor.ParseInt;
+import org.supercsv.cellprocessor.ParseLong;
+import org.supercsv.cellprocessor.constraint.DMinMax;
+import org.supercsv.cellprocessor.constraint.Equals;
+import org.supercsv.cellprocessor.constraint.ForbidSubStr;
+import org.supercsv.cellprocessor.constraint.IsIncludedIn;
+import org.supercsv.cellprocessor.constraint.LMinMax;
+import org.supercsv.cellprocessor.constraint.NotNull;
+import org.supercsv.cellprocessor.constraint.RequireHashCode;
+import org.supercsv.cellprocessor.constraint.RequireSubStr;
+import org.supercsv.cellprocessor.constraint.StrMinMax;
+import org.supercsv.cellprocessor.constraint.StrNotNullOrEmpty;
+import org.supercsv.cellprocessor.constraint.StrRegEx;
+import org.supercsv.cellprocessor.constraint.Strlen;
+import org.supercsv.cellprocessor.constraint.Unique;
+import org.supercsv.cellprocessor.constraint.UniqueHashCode;
+import org.supercsv.cellprocessor.ift.CellProcessor;
+import org.supercsv.exception.SuperCsvCellProcessorException;
+import org.supercsv.io.CsvListReader;
+import org.supercsv.prefs.CsvPreference;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"csv", "schema", "validation"})
+@CapabilityDescription("Validates the contents of FlowFiles against a 
user-specified CSV schema. " +
+"Take a look at the additional documentation of this processor for 
some schema examples.")
+public class ValidateCsv extends AbstractProcessor {
+
+private final static List 

[GitHub] nifi pull request #476: NIFI-1942 Processor to validate CSV against user-sup...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/476#discussion_r72541263
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateCsv.java
 ---
@@ -0,0 +1,613 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.Reader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.supercsv.cellprocessor.Optional;
+import org.supercsv.cellprocessor.ParseBigDecimal;
+import org.supercsv.cellprocessor.ParseBool;
+import org.supercsv.cellprocessor.ParseChar;
+import org.supercsv.cellprocessor.ParseDate;
+import org.supercsv.cellprocessor.ParseDouble;
+import org.supercsv.cellprocessor.ParseInt;
+import org.supercsv.cellprocessor.ParseLong;
+import org.supercsv.cellprocessor.constraint.DMinMax;
+import org.supercsv.cellprocessor.constraint.Equals;
+import org.supercsv.cellprocessor.constraint.ForbidSubStr;
+import org.supercsv.cellprocessor.constraint.IsIncludedIn;
+import org.supercsv.cellprocessor.constraint.LMinMax;
+import org.supercsv.cellprocessor.constraint.NotNull;
+import org.supercsv.cellprocessor.constraint.RequireHashCode;
+import org.supercsv.cellprocessor.constraint.RequireSubStr;
+import org.supercsv.cellprocessor.constraint.StrMinMax;
+import org.supercsv.cellprocessor.constraint.StrNotNullOrEmpty;
+import org.supercsv.cellprocessor.constraint.StrRegEx;
+import org.supercsv.cellprocessor.constraint.Strlen;
+import org.supercsv.cellprocessor.constraint.Unique;
+import org.supercsv.cellprocessor.constraint.UniqueHashCode;
+import org.supercsv.cellprocessor.ift.CellProcessor;
+import org.supercsv.exception.SuperCsvCellProcessorException;
+import org.supercsv.io.CsvListReader;
+import org.supercsv.prefs.CsvPreference;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"csv", "schema", "validation"})
+@CapabilityDescription("Validates the contents of FlowFiles against a 
user-specified CSV schema. " +
+"Take a look at the additional documentation of this processor for 
some schema examples.")
+public class ValidateCsv extends AbstractProcessor {
+
+private final static List 

[GitHub] nifi pull request #476: NIFI-1942 Processor to validate CSV against user-sup...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/476#discussion_r72542364
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateCsv.java
 ---
@@ -0,0 +1,613 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.Reader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.supercsv.cellprocessor.Optional;
+import org.supercsv.cellprocessor.ParseBigDecimal;
+import org.supercsv.cellprocessor.ParseBool;
+import org.supercsv.cellprocessor.ParseChar;
+import org.supercsv.cellprocessor.ParseDate;
+import org.supercsv.cellprocessor.ParseDouble;
+import org.supercsv.cellprocessor.ParseInt;
+import org.supercsv.cellprocessor.ParseLong;
+import org.supercsv.cellprocessor.constraint.DMinMax;
+import org.supercsv.cellprocessor.constraint.Equals;
+import org.supercsv.cellprocessor.constraint.ForbidSubStr;
+import org.supercsv.cellprocessor.constraint.IsIncludedIn;
+import org.supercsv.cellprocessor.constraint.LMinMax;
+import org.supercsv.cellprocessor.constraint.NotNull;
+import org.supercsv.cellprocessor.constraint.RequireHashCode;
+import org.supercsv.cellprocessor.constraint.RequireSubStr;
+import org.supercsv.cellprocessor.constraint.StrMinMax;
+import org.supercsv.cellprocessor.constraint.StrNotNullOrEmpty;
+import org.supercsv.cellprocessor.constraint.StrRegEx;
+import org.supercsv.cellprocessor.constraint.Strlen;
+import org.supercsv.cellprocessor.constraint.Unique;
+import org.supercsv.cellprocessor.constraint.UniqueHashCode;
+import org.supercsv.cellprocessor.ift.CellProcessor;
+import org.supercsv.exception.SuperCsvCellProcessorException;
+import org.supercsv.io.CsvListReader;
+import org.supercsv.prefs.CsvPreference;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"csv", "schema", "validation"})
+@CapabilityDescription("Validates the contents of FlowFiles against a 
user-specified CSV schema. " +
+"Take a look at the additional documentation of this processor for 
some schema examples.")
+public class ValidateCsv extends AbstractProcessor {
+
+private final static List 

[GitHub] nifi pull request #563: NIFI-2078, 2363, 2364: External state management. CL...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72559265
  
--- Diff: 
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/pom.xml ---
@@ -35,6 +35,10 @@
 nifi-utils
 
 
+org.apache.nifi
+nifi-expression-language
--- End diff --

I added EL dependency to use StandardPropertyValue class at 
ConsumeKafka.onPropertyModified.

Since ConsumeKafka allows EL for properties that needed to get state from 
Kafka, EL has to be evaluated at onPropertyModified, too. But to do so, we need 
PropertyValue instance, which is not available at onPropertyModified because it 
only receives String representation of oldValue and newValue. An implementation 
class of PropertyValue is not included in nifi-api. So, I needed to add the EL 
dependency.

However, I felt it's a little bit strange to support EL for things like 
'topic' or 'bootstrap_servers', since ConsumeKafka doesn't take input flow 
files, and these properties can not be changed after ConsumerKafka connects to 
Kafka. It keep using the same kafkaResource instance. So, those property values 
are more static, rather than dynamically evaluated. It makes sense to support 
EL for those values for PublishKafka, though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396834#comment-15396834
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72559265
  
--- Diff: 
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/pom.xml ---
@@ -35,6 +35,10 @@
 nifi-utils
 
 
+org.apache.nifi
+nifi-expression-language
--- End diff --

I added EL dependency to use StandardPropertyValue class at 
ConsumeKafka.onPropertyModified.

Since ConsumeKafka allows EL for properties that needed to get state from 
Kafka, EL has to be evaluated at onPropertyModified, too. But to do so, we need 
PropertyValue instance, which is not available at onPropertyModified because it 
only receives String representation of oldValue and newValue. An implementation 
class of PropertyValue is not included in nifi-api. So, I needed to add the EL 
dependency.

However, I felt it's a little bit strange to support EL for things like 
'topic' or 'bootstrap_servers', since ConsumeKafka doesn't take input flow 
files, and these properties can not be changed after ConsumerKafka connects to 
Kafka. It keep using the same kafkaResource instance. So, those property values 
are more static, rather than dynamically evaluated. It makes sense to support 
EL for those values for PublishKafka, though.


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396843#comment-15396843
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72559943
  
--- Diff: 
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/pom.xml ---
@@ -35,6 +35,10 @@
 nifi-utils
 
 
+org.apache.nifi
+nifi-expression-language
--- End diff --

BOOTSTRAP_SERVERS, TOPIC, CLIENT_ID are the properties shared among 
Consume/ProduceKafka. If we can separate and let Consume, ProduceKafka have 
those as its own property, ProduceKafka to support EL but not ConsumeKafka, 
then the dependency can be removed.

But existing flow configuration may have to be updated if one uses EL for 
those properties of ConsumeKafka. 

@olegz @JPercivall How do you think?


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #563: NIFI-2078, 2363, 2364: External state management. CL...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72559943
  
--- Diff: 
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/pom.xml ---
@@ -35,6 +35,10 @@
 nifi-utils
 
 
+org.apache.nifi
+nifi-expression-language
--- End diff --

BOOTSTRAP_SERVERS, TOPIC, CLIENT_ID are the properties shared among 
Consume/ProduceKafka. If we can separate and let Consume, ProduceKafka have 
those as its own property, ProduceKafka to support EL but not ConsumeKafka, 
then the dependency can be removed.

But existing flow configuration may have to be updated if one uses EL for 
those properties of ConsumeKafka. 

@olegz @JPercivall How do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-2415) GenerateTableFetch/QueryDatabaseTable cannot store state

[Haimo Liu (JIRA)]

Haimo Liu created NIFI-2415:
---

 Summary: GenerateTableFetch/QueryDatabaseTable cannot store state
 Key: NIFI-2415
 URL: https://issues.apache.org/jira/browse/NIFI-2415
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Affects Versions: 1.0.0
Reporter: Haimo Liu
Assignee: Matt Burgess
Priority: Critical
 Fix For: 1.0.0
 Attachments: Screen Shot 2016-07-27 at 11.10.45 PM.png

running NIFI in a clustered mode, 7/27 build

Both GenerateTableFetch and QueryDatabaseTable cannot store state in a 
clustered mode (checked ListFile, can store state successfully)

see attached screenshot



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2415) GenerateTableFetch/QueryDatabaseTable cannot store state

[Haimo Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haimo Liu updated NIFI-2415:

Attachment: Screen Shot 2016-07-27 at 11.10.45 PM.png

> GenerateTableFetch/QueryDatabaseTable cannot store state
> 
>
> Key: NIFI-2415
> URL: https://issues.apache.org/jira/browse/NIFI-2415
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Assignee: Matt Burgess
>Priority: Critical
> Fix For: 1.0.0
>
> Attachments: Screen Shot 2016-07-27 at 11.10.45 PM.png
>
>
> running NIFI in a clustered mode, 7/27 build
> Both GenerateTableFetch and QueryDatabaseTable cannot store state in a 
> clustered mode (checked ListFile, can store state successfully)
> see attached screenshot



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2415) GenerateTableFetch/QueryDatabaseTable cannot store state

[Haimo Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haimo Liu updated NIFI-2415:

Description: 
running NIFI in a clustered mode, 7/27 build

Both GenerateTableFetch and QueryDatabaseTable cannot store state in a 
clustered mode (checked ListFile, can store state successfully)

see attached screenshot, trace below:

2016-07-27 23:25:00,014 INFO [ProcessThread(sid:1 cport:-1):] 
o.a.z.server.PrepRequestProcessor Got user-level KeeperException when 
processing sessionid:0x2562f7875130009 type:create cxid:0xe4d zxid:0x73733 
txntype:-1 reqpath:n/a Error 
Path:/nifi/components/2f810bb3-0156-1000--16128f8b 
Error:KeeperErrorCode = InvalidACL for 
/nifi/components/2f810bb3-0156-1000--16128f8b
2016-07-27 23:25:00,017 WARN [Framework Task Thread Thread-1] 
o.a.n.c.s.m.StandardStateManagerProvider Component with ID {} was removed from 
NiFi instance but failed to clear clustered state for the component
java.io.IOException: Failed to set cluster-wide state in ZooKeeper for 
component with ID 2f810bb3-0156-1000--16128f8b
at 
org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:343)
 ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283)
 ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228)
 ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.clear(ZooKeeperStateProvider.java:453)
 ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.state.StandardStateManager.clear(StandardStateManager.java:85)
 ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.state.manager.StandardStateManagerProvider.onComponentRemoved(StandardStateManagerProvider.java:276)
 ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.groups.StandardProcessGroup$1.run(StandardProcessGroup.java:730)
 [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_77]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[na:1.8.0_77]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 [na:1.8.0_77]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 [na:1.8.0_77]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_77]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: 
KeeperErrorCode = InvalidACL for 
/nifi/components/2f810bb3-0156-1000--16128f8b
at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:121) 
~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) 
~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) 
~[zookeeper-3.4.6.jar:3.4.6-1569965]
at 
org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
 ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
 ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
... 13 common frames omitted

  was:
running NIFI in a clustered mode, 7/27 build

Both GenerateTableFetch and QueryDatabaseTable cannot store state in a 
clustered mode (checked ListFile, can store state successfully)

see attached screenshot


> GenerateTableFetch/QueryDatabaseTable cannot store state
> 
>
> Key: NIFI-2415
> URL: https://issues.apache.org/jira/browse/NIFI-2415
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Assignee: Matt Burgess
>Priority: Critical
> Fix For: 1.0.0
>
> Attachments: Screen Shot 2016-07-27 at 11.10.45 PM.png
>
>
> running NIFI in a clustered mode, 7/27 build
> Both GenerateTableFetch and QueryDatabaseTable cannot store state in a 
> clustered mode (checked 

[jira] [Commented] (NIFI-2304) Cluster Coordinator reported incorrectly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396867#comment-15396867
 ] 

ASF GitHub Bot commented on NIFI-2304:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/688#discussion_r72561569
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/ClusterProtocolHeartbeatMonitor.java
 ---
@@ -91,12 +91,8 @@ public ClusterProtocolHeartbeatMonitor(final 
ClusterCoordinator clusterCoordinat
 this.clusterNodesPath = 
zkClientConfig.resolvePath("cluster/nodes");
 
 String hostname = 
properties.getProperty(NiFiProperties.CLUSTER_NODE_ADDRESS);
-if (hostname == null) {
-try {
-hostname = InetAddress.getLocalHost().getHostName();
-} catch (UnknownHostException e) {
-throw new RuntimeException("Unable to determine local 
hostname and the '" + NiFiProperties.CLUSTER_NODE_ADDRESS + "' property is not 
set");
-}
+if (hostname == null || hostname.trim().isEmpty()) {
+hostname = "localhost";
--- End diff --

Is there any specific reason to change from 
`InetAddress.getLocalHost().getHostName()` to `"localhost"`?

I thought if server's hostname is properly configured at OS level, 
administrator doesn't have to set hostname in nifi.properties, which makes 
configuration management easier since they can use the same properties file 
among nodes.


> Cluster Coordinator reported incorrectly
> 
>
> Key: NIFI-2304
> URL: https://issues.apache.org/jira/browse/NIFI-2304
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Bryan Bende
>Assignee: Mark Payne
>Priority: Minor
> Fix For: 1.0.0
>
>
> I created a two node cluster locally, running embedded ZK on node1.. both 
> nodes start up, but when trying to access the UI on the second node, it says 
> no coordinator exists. 
> Looking in the logs the coordinator is being reported as ":8889" where 8889 
> is the node protocol port. I left the node host blank assuming it would 
> default to localhost, but looks like it isn't. When I filled in the node host 
> everything worked as expected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #688: NIFI-2304: If node address is not set, default to lo...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/688#discussion_r72561569
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/ClusterProtocolHeartbeatMonitor.java
 ---
@@ -91,12 +91,8 @@ public ClusterProtocolHeartbeatMonitor(final 
ClusterCoordinator clusterCoordinat
 this.clusterNodesPath = 
zkClientConfig.resolvePath("cluster/nodes");
 
 String hostname = 
properties.getProperty(NiFiProperties.CLUSTER_NODE_ADDRESS);
-if (hostname == null) {
-try {
-hostname = InetAddress.getLocalHost().getHostName();
-} catch (UnknownHostException e) {
-throw new RuntimeException("Unable to determine local 
hostname and the '" + NiFiProperties.CLUSTER_NODE_ADDRESS + "' property is not 
set");
-}
+if (hostname == null || hostname.trim().isEmpty()) {
+hostname = "localhost";
--- End diff --

Is there any specific reason to change from 
`InetAddress.getLocalHost().getHostName()` to `"localhost"`?

I thought if server's hostname is properly configured at OS level, 
administrator doesn't have to set hostname in nifi.properties, which makes 
configuration management easier since they can use the same properties file 
among nodes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2304) Cluster Coordinator reported incorrectly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396872#comment-15396872
 ] 

ASF GitHub Bot commented on NIFI-2304:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/688#discussion_r72561906
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/ClusterProtocolHeartbeatMonitor.java
 ---
@@ -91,12 +91,8 @@ public ClusterProtocolHeartbeatMonitor(final 
ClusterCoordinator clusterCoordinat
 this.clusterNodesPath = 
zkClientConfig.resolvePath("cluster/nodes");
 
 String hostname = 
properties.getProperty(NiFiProperties.CLUSTER_NODE_ADDRESS);
-if (hostname == null) {
-try {
-hostname = InetAddress.getLocalHost().getHostName();
-} catch (UnknownHostException e) {
-throw new RuntimeException("Unable to determine local 
hostname and the '" + NiFiProperties.CLUSTER_NODE_ADDRESS + "' property is not 
set");
-}
+if (hostname == null || hostname.trim().isEmpty()) {
--- End diff --

There's another similar null check against port `if (port == null) {`. I 
think it should be addressed, too.
We may be able to take advantage of our existing StringUtils.isBlank().


> Cluster Coordinator reported incorrectly
> 
>
> Key: NIFI-2304
> URL: https://issues.apache.org/jira/browse/NIFI-2304
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Bryan Bende
>Assignee: Mark Payne
>Priority: Minor
> Fix For: 1.0.0
>
>
> I created a two node cluster locally, running embedded ZK on node1.. both 
> nodes start up, but when trying to access the UI on the second node, it says 
> no coordinator exists. 
> Looking in the logs the coordinator is being reported as ":8889" where 8889 
> is the node protocol port. I left the node host blank assuming it would 
> default to localhost, but looks like it isn't. When I filled in the node host 
> everything worked as expected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #688: NIFI-2304: If node address is not set, default to lo...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/688#discussion_r72561906
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/ClusterProtocolHeartbeatMonitor.java
 ---
@@ -91,12 +91,8 @@ public ClusterProtocolHeartbeatMonitor(final 
ClusterCoordinator clusterCoordinat
 this.clusterNodesPath = 
zkClientConfig.resolvePath("cluster/nodes");
 
 String hostname = 
properties.getProperty(NiFiProperties.CLUSTER_NODE_ADDRESS);
-if (hostname == null) {
-try {
-hostname = InetAddress.getLocalHost().getHostName();
-} catch (UnknownHostException e) {
-throw new RuntimeException("Unable to determine local 
hostname and the '" + NiFiProperties.CLUSTER_NODE_ADDRESS + "' property is not 
set");
-}
+if (hostname == null || hostname.trim().isEmpty()) {
--- End diff --

There's another similar null check against port `if (port == null) {`. I 
think it should be addressed, too.
We may be able to take advantage of our existing StringUtils.isBlank().


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396881#comment-15396881
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72562357
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/ClusterProtocolHeartbeatMonitor.java
 ---
@@ -91,7 +92,7 @@ public ClusterProtocolHeartbeatMonitor(final 
ClusterCoordinator clusterCoordinat
 this.clusterNodesPath = 
zkClientConfig.resolvePath("cluster/nodes");
 
 String hostname = 
properties.getProperty(NiFiProperties.CLUSTER_NODE_ADDRESS);
-if (hostname == null) {
+if (StringUtils.isEmpty(hostname)) {
--- End diff --

Thanks for pointing this out. I will remove these changes from this PR. 
Posted my comments on #688.


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #563: NIFI-2078, 2363, 2364: External state management. CL...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72562357
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/ClusterProtocolHeartbeatMonitor.java
 ---
@@ -91,7 +92,7 @@ public ClusterProtocolHeartbeatMonitor(final 
ClusterCoordinator clusterCoordinat
 this.clusterNodesPath = 
zkClientConfig.resolvePath("cluster/nodes");
 
 String hostname = 
properties.getProperty(NiFiProperties.CLUSTER_NODE_ADDRESS);
-if (hostname == null) {
+if (StringUtils.isEmpty(hostname)) {
--- End diff --

Thanks for pointing this out. I will remove these changes from this PR. 
Posted my comments on #688.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

[Brian Davis (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395708#comment-15395708
 ] 

Brian Davis commented on NIFI-2395:
---

I forgot to keep the stacktrace.  I am waiting on the nifi to hit the issue 
again on my system.  It should be within the day.  Until then here is what I 
remember:

12 threads at   
[org.apache.nifi.provenance.PersistentProvanceRepository.persistRecord(PersistentProvanceRepository:661)
 
|https://github.com/apache/nifi/blob/nifi-0.6.0/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L661]
1 thread at 
[org.apache.nifi.provenance.PersistentProvanceRepository.rollover(PersistentProvanceRepository:1322)
 
|https://github.com/apache/nifi/blob/nifi-0.6.0/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1322]
1 thread at 
[org.apache.nifi.provenance.PersistentProvanceRepository.mergeJournals(PersistentProvanceRepository:1662)
 
|https://github.com/apache/nifi/blob/nifi-0.6.0/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1662]

How they got to these points I will have to wait for the issue to come back.


> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Blocker
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} bytes)", 
> journalFileCount, repoSize, journalCountThreshold, sizeThreshold);
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> }
> logger.info("Provenance Repository has now caught up with 
> rolling over journal files. Current number of "
> + "journal files to be rolled over is {}", 
> journalFileCount);
> }
> {code}
> My nifi is at the sleep indefinitely.  The reason my nifi cannot move forward 
> is because of the thread doing the merge is stopped.  The thread doing the 
> merge is at:
> {code}
> accepted = eventQueue.offer(new Tuple<>(record, blockIndex), 10, 
> TimeUnit.MILLISECONDS);
> {code}
> so the queue is full.  
> What I believe happened is that the callables 

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396758#comment-15396758
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user brosander commented on the issue:

https://github.com/apache/nifi/pull/695
  
@alopresto I agree with your concerns about the different entry points, 
I'll try to unify them and get down to a single shell script with the main 
method and argument parsing determining the action


> Command Line Keystore and Truststore utility
> 
>
> Key: NIFI-2193
> URL: https://issues.apache.org/jira/browse/NIFI-2193
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Bryan Rosander
>Assignee: Bryan Rosander
>
> In order to facilitate secure setup of NiFi, it would be useful to have a 
> command line utility capable of generating the required keystores, 
> truststore, and relevant configuration files.
> It should be able to generate keystores for each NiFi node, a truststore that 
> they all use, and relevant passwords and configuration files for using the 
> keystores and truststore.
> Additionally, in order to support distributed deployment, a web based 
> certificate authority with corresponding client will allow for each NiFi 
> instance to generate its own keypair and then request signing by the CA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[brosander]

Github user brosander commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72551830
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/util/TlsHelper.java
 ---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.util;
+
+import org.apache.nifi.security.util.CertificateUtils;
+import org.apache.nifi.toolkit.tls.commandLine.TlsToolkitCommandLine;
+import org.apache.nifi.toolkit.tls.configuration.TlsHelperConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.cert.crmf.CRMFException;
+import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter;
+import org.bouncycastle.cert.jcajce.JcaX509ExtensionUtils;
+import org.bouncycastle.eac.EACException;
+import org.bouncycastle.jce.provider.BouncyCastleProvider;
+import org.bouncycastle.openssl.PEMParser;
+import org.bouncycastle.openssl.jcajce.JcaMiscPEMGenerator;
+import org.bouncycastle.operator.OperatorCreationException;
+import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequestBuilder;
+import org.bouncycastle.util.io.pem.PemWriter;
+
+import javax.crypto.Mac;
+import javax.crypto.spec.SecretKeySpec;
+import javax.security.auth.x500.X500Principal;
+import java.io.IOException;
+import java.io.StringReader;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.security.GeneralSecurityException;
+import java.security.InvalidKeyException;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.PublicKey;
+import java.security.Security;
+import java.security.SignatureException;
+import java.security.cert.Certificate;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.security.spec.InvalidKeySpecException;
+
+public class TlsHelper {
+public static final String PROVIDER = 
BouncyCastleProvider.PROVIDER_NAME;
+public static final String PKCS12 = "PKCS12";
+private final KeyPairGenerator keyPairGenerator;
+private final int days;
+private final String signingAlgorithm;
+
+public TlsHelper(TlsHelperConfig tlsHelperConfig) throws 
NoSuchAlgorithmException {
+this(tlsHelperConfig.getDays(), tlsHelperConfig.getKeySize(), 
tlsHelperConfig.getKeyPairAlgorithm(), tlsHelperConfig.getSigningAlgorithm());
+}
+
+public TlsHelper(TlsToolkitCommandLine tlsToolkitCommandLine) throws 
NoSuchAlgorithmException {
+this(tlsToolkitCommandLine.getTlsHelperConfig());
+}
+
+public TlsHelper(int days, int keySize, String keyPairAlgorithm, 
String signingAlgorithm) throws NoSuchAlgorithmException {
+this(createKeyPairGenerator(keyPairAlgorithm, keySize), days, 
signingAlgorithm);
+}
+
+protected TlsHelper(KeyPairGenerator keyPairGenerator, int days, 
String signingAlgorithm) {
+this.keyPairGenerator = keyPairGenerator;
+this.days = days;
+this.signingAlgorithm = signingAlgorithm;
+}
+
+public static void addBouncyCastleProvider() {
+Security.addProvider(new BouncyCastleProvider());
+}
+
+private static KeyPairGenerator createKeyPairGenerator(String 
algorithm, int keySize) throws NoSuchAlgorithmException {
+KeyPairGenerator instance = 
KeyPairGenerator.getInstance(algorithm);
+instance.initialize(keySize);
+return instance;
+}
  

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396754#comment-15396754
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user brosander commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72551782
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/util/TlsHelper.java
 ---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.util;
+
+import org.apache.nifi.security.util.CertificateUtils;
+import org.apache.nifi.toolkit.tls.commandLine.TlsToolkitCommandLine;
+import org.apache.nifi.toolkit.tls.configuration.TlsHelperConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.cert.crmf.CRMFException;
+import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter;
+import org.bouncycastle.cert.jcajce.JcaX509ExtensionUtils;
+import org.bouncycastle.eac.EACException;
+import org.bouncycastle.jce.provider.BouncyCastleProvider;
+import org.bouncycastle.openssl.PEMParser;
+import org.bouncycastle.openssl.jcajce.JcaMiscPEMGenerator;
+import org.bouncycastle.operator.OperatorCreationException;
+import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequestBuilder;
+import org.bouncycastle.util.io.pem.PemWriter;
+
+import javax.crypto.Mac;
+import javax.crypto.spec.SecretKeySpec;
+import javax.security.auth.x500.X500Principal;
+import java.io.IOException;
+import java.io.StringReader;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.security.GeneralSecurityException;
+import java.security.InvalidKeyException;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.PublicKey;
+import java.security.Security;
+import java.security.SignatureException;
+import java.security.cert.Certificate;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.security.spec.InvalidKeySpecException;
+
+public class TlsHelper {
+public static final String PROVIDER = 
BouncyCastleProvider.PROVIDER_NAME;
+public static final String PKCS12 = "PKCS12";
+private final KeyPairGenerator keyPairGenerator;
+private final int days;
+private final String signingAlgorithm;
+
+public TlsHelper(TlsHelperConfig tlsHelperConfig) throws 
NoSuchAlgorithmException {
+this(tlsHelperConfig.getDays(), tlsHelperConfig.getKeySize(), 
tlsHelperConfig.getKeyPairAlgorithm(), tlsHelperConfig.getSigningAlgorithm());
+}
+
+public TlsHelper(TlsToolkitCommandLine tlsToolkitCommandLine) throws 
NoSuchAlgorithmException {
+this(tlsToolkitCommandLine.getTlsHelperConfig());
+}
+
+public TlsHelper(int days, int keySize, String keyPairAlgorithm, 
String signingAlgorithm) throws NoSuchAlgorithmException {
+this(createKeyPairGenerator(keyPairAlgorithm, keySize), days, 
signingAlgorithm);
+}
+
+protected TlsHelper(KeyPairGenerator keyPairGenerator, int days, 
String signingAlgorithm) {
+this.keyPairGenerator = keyPairGenerator;
+this.days = days;
+this.signingAlgorithm = signingAlgorithm;
+}
+
+public static void addBouncyCastleProvider() {
+Security.addProvider(new BouncyCastleProvider());
+}
+
+private static KeyPairGenerator 

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[brosander]

Github user brosander commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72551782
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/util/TlsHelper.java
 ---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.util;
+
+import org.apache.nifi.security.util.CertificateUtils;
+import org.apache.nifi.toolkit.tls.commandLine.TlsToolkitCommandLine;
+import org.apache.nifi.toolkit.tls.configuration.TlsHelperConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.cert.crmf.CRMFException;
+import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter;
+import org.bouncycastle.cert.jcajce.JcaX509ExtensionUtils;
+import org.bouncycastle.eac.EACException;
+import org.bouncycastle.jce.provider.BouncyCastleProvider;
+import org.bouncycastle.openssl.PEMParser;
+import org.bouncycastle.openssl.jcajce.JcaMiscPEMGenerator;
+import org.bouncycastle.operator.OperatorCreationException;
+import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequestBuilder;
+import org.bouncycastle.util.io.pem.PemWriter;
+
+import javax.crypto.Mac;
+import javax.crypto.spec.SecretKeySpec;
+import javax.security.auth.x500.X500Principal;
+import java.io.IOException;
+import java.io.StringReader;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.security.GeneralSecurityException;
+import java.security.InvalidKeyException;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.PublicKey;
+import java.security.Security;
+import java.security.SignatureException;
+import java.security.cert.Certificate;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.security.spec.InvalidKeySpecException;
+
+public class TlsHelper {
+public static final String PROVIDER = 
BouncyCastleProvider.PROVIDER_NAME;
+public static final String PKCS12 = "PKCS12";
+private final KeyPairGenerator keyPairGenerator;
+private final int days;
+private final String signingAlgorithm;
+
+public TlsHelper(TlsHelperConfig tlsHelperConfig) throws 
NoSuchAlgorithmException {
+this(tlsHelperConfig.getDays(), tlsHelperConfig.getKeySize(), 
tlsHelperConfig.getKeyPairAlgorithm(), tlsHelperConfig.getSigningAlgorithm());
+}
+
+public TlsHelper(TlsToolkitCommandLine tlsToolkitCommandLine) throws 
NoSuchAlgorithmException {
+this(tlsToolkitCommandLine.getTlsHelperConfig());
+}
+
+public TlsHelper(int days, int keySize, String keyPairAlgorithm, 
String signingAlgorithm) throws NoSuchAlgorithmException {
+this(createKeyPairGenerator(keyPairAlgorithm, keySize), days, 
signingAlgorithm);
+}
+
+protected TlsHelper(KeyPairGenerator keyPairGenerator, int days, 
String signingAlgorithm) {
+this.keyPairGenerator = keyPairGenerator;
+this.days = days;
+this.signingAlgorithm = signingAlgorithm;
+}
+
+public static void addBouncyCastleProvider() {
+Security.addProvider(new BouncyCastleProvider());
+}
+
+private static KeyPairGenerator createKeyPairGenerator(String 
algorithm, int keySize) throws NoSuchAlgorithmException {
+KeyPairGenerator instance = 
KeyPairGenerator.getInstance(algorithm);
+instance.initialize(keySize);
+return instance;
+}
  

[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396807#comment-15396807
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72557366
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ClearComponentStateEndpointMerger.java
 ---
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.cluster.coordination.http.endpoints;
+
+import org.apache.nifi.cluster.manager.NodeResponse;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.web.api.entity.ClearComponentStateResultEntity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URI;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+public class ClearComponentStateEndpointMerger extends 
AbstractSingleEntityEndpoint {
+public static final Pattern PROCESSOR_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/processors/[a-f0-9\\-]{36}/state/clear-requests");
+public static final Pattern CONTROLLER_SERVICE_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/controller-services/[a-f0-9\\-]{36}/state/clear-requests");
+public static final Pattern REPORTING_TASK_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/reporting-tasks/[a-f0-9\\-]{36}/state/clear-requests");
--- End diff --

Yes, that's how it's done today. There may be some way to get URI patterns 
align with corresponding Resource class methods, though.


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #563: NIFI-2078, 2363, 2364: External state management. CL...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72557366
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/ClearComponentStateEndpointMerger.java
 ---
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.cluster.coordination.http.endpoints;
+
+import org.apache.nifi.cluster.manager.NodeResponse;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.web.api.entity.ClearComponentStateResultEntity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URI;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+public class ClearComponentStateEndpointMerger extends 
AbstractSingleEntityEndpoint {
+public static final Pattern PROCESSOR_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/processors/[a-f0-9\\-]{36}/state/clear-requests");
+public static final Pattern CONTROLLER_SERVICE_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/controller-services/[a-f0-9\\-]{36}/state/clear-requests");
+public static final Pattern REPORTING_TASK_STATE_URI_PATTERN = 
Pattern.compile("/nifi-api/reporting-tasks/[a-f0-9\\-]{36}/state/clear-requests");
--- End diff --

Yes, that's how it's done today. There may be some way to get URI patterns 
align with corresponding Resource class methods, though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2367) Overlapping text in Cluster pane after error

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395578#comment-15395578
 ] 

ASF GitHub Bot commented on NIFI-2367:
--

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/715#discussion_r72430022
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-canvas-utils.js
 ---
@@ -516,10 +516,12 @@ nf.CanvasUtils = (function () {
 });
 
 // update the tooltip background
-
selection.select('rect.bulletin-background').classed('has-bulletins', true);
+selection.select('text.bulletin-icon').style("visibility", 
"visible");
+
selection.select('rect.bulletin-background').style("visibility", 
"visible").classed('has-bulletins', true);
--- End diff --

With this proposed change we could remove the has-bulletins class and just 
integrate those styles into the base style of the bulletin-background since 
we're only ever showing it when it has bulletins.


> Overlapping text in Cluster pane after error
> 
>
> Key: NIFI-2367
> URL: https://issues.apache.org/jira/browse/NIFI-2367
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Joseph Percivall
>Assignee: Scott Aslan
> Attachments: Screen Shot 2016-07-22 at 2.42.48 PM.png
>
>
> 3 node secure cluster
> In cluster pane
> hit 500 error when refreshing pane (output in developer tools): 
> https://localhost:8481/nifi-api/controller/cluster Failed to load resource: 
> the server responded with a status of 500 (Internal Server Error)
> See overlapping text of "fix" and "home" in resulting pane (see attached 
> screenshot).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

[Brian Davis (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395657#comment-15395657
 ] 

Brian Davis edited comment on NIFI-2395 at 7/27/16 1:22 PM:


My configuration for the provenance should be very close to the default.  The 
only difference is I am using 2 separate disks to do provenance and only 
provenance.  Here is the config:

{noformat}
# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository

# Persistent Provenance repository Properties
nifi.provenance.repository.directory.prov1=/data/04/nifi/provenance_repository
nifi.provenance.repository.directory.prov2=/data/05/nifi/provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.indexing.threads=1
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
# Comma-separated list of fields. Fields that are not indexed will not be 
searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, 
AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, 
ProcessorID
# FlowFile Attributes that should be indexed and made searchable
nifi.provenance.repository.indexed.attributes=twitter.msg, language
# Large values for the shard size will result in more Java heap usage when 
searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving 
a Provenance Event form
# the repository.  If the length of any attribute exceeds this value, it will 
be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536

# Volatile Provenance Repository Properties
nifi.provenance.repository.buffer.size=10
{noformat}

Now that I have gone through these properties with a little more detail, I see 
some places that need to change.  :)


was (Author: badavis):
My configuration for the provenance should be very close to the default.  The 
only difference is I am using 2 separate disks to do provenance and only 
provenance.  Here is the config:

# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository

# Persistent Provenance repository Properties
nifi.provenance.repository.directory.prov1=/data/04/nifi/provenance_repository
nifi.provenance.repository.directory.prov2=/data/05/nifi/provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.indexing.threads=1
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
# Comma-separated list of fields. Fields that are not indexed will not be 
searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, 
AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, 
ProcessorID
# FlowFile Attributes that should be indexed and made searchable
nifi.provenance.repository.indexed.attributes=twitter.msg, language
# Large values for the shard size will result in more Java heap usage when 
searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving 
a Provenance Event form
# the repository.  If the length of any attribute exceeds this value, it will 
be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536

# Volatile Provenance Repository Properties
nifi.provenance.repository.buffer.size=10


Now that I have gone through these properties with a little more detail, I see 
some places that need to change.  :)

> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Blocker
>

[jira] [Commented] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

[Brian Davis (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395657#comment-15395657
 ] 

Brian Davis commented on NIFI-2395:
---

My configuration for the provenance should be very close to the default.  The 
only difference is I am using 2 separate disks to do provenance and only 
provenance.  Here is the config:

# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository

# Persistent Provenance repository Properties
nifi.provenance.repository.directory.prov1=/data/04/nifi/provenance_repository
nifi.provenance.repository.directory.prov2=/data/05/nifi/provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.indexing.threads=1
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
# Comma-separated list of fields. Fields that are not indexed will not be 
searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, 
AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, 
ProcessorID
# FlowFile Attributes that should be indexed and made searchable
nifi.provenance.repository.indexed.attributes=twitter.msg, language
# Large values for the shard size will result in more Java heap usage when 
searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving 
a Provenance Event form
# the repository.  If the length of any attribute exceeds this value, it will 
be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536

# Volatile Provenance Repository Properties
nifi.provenance.repository.buffer.size=10


Now that I have gone through these properties with a little more detail, I see 
some places that need to change.  :)

> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Blocker
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} 

[jira] [Updated] (NIFI-1969) Update Documentation for new Authorization Properties

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-1969:
--
Labels: migration  (was: )

> Update Documentation for new Authorization Properties
> -
>
> Key: NIFI-1969
> URL: https://issues.apache.org/jira/browse/NIFI-1969
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Andrew Lim
>Priority: Blocker
>  Labels: migration
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1969) Update Documentation for new Authorization Properties

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-1969:
--
Priority: Critical  (was: Blocker)

> Update Documentation for new Authorization Properties
> -
>
> Key: NIFI-1969
> URL: https://issues.apache.org/jira/browse/NIFI-1969
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Andrew Lim
>Priority: Critical
>  Labels: migration
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #695: NIFI-2193 - Command line SSL config utility as well as cert...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/695
  
I am reviewing, focusing on the cryptographic code in here as well as the 
token signing for communication between the nodes and CA over an untrusted 
channel. 

@brosander can you please provide some documentation/admin guide? The only 
thing I have seen so far is the usage in the NiFiToolkitTLS class. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #476: NIFI-1942 Processor to validate CSV against user-sup...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/476#discussion_r72538224
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateCsv.java
 ---
@@ -0,0 +1,613 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.Reader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.supercsv.cellprocessor.Optional;
+import org.supercsv.cellprocessor.ParseBigDecimal;
+import org.supercsv.cellprocessor.ParseBool;
+import org.supercsv.cellprocessor.ParseChar;
+import org.supercsv.cellprocessor.ParseDate;
+import org.supercsv.cellprocessor.ParseDouble;
+import org.supercsv.cellprocessor.ParseInt;
+import org.supercsv.cellprocessor.ParseLong;
+import org.supercsv.cellprocessor.constraint.DMinMax;
+import org.supercsv.cellprocessor.constraint.Equals;
+import org.supercsv.cellprocessor.constraint.ForbidSubStr;
+import org.supercsv.cellprocessor.constraint.IsIncludedIn;
+import org.supercsv.cellprocessor.constraint.LMinMax;
+import org.supercsv.cellprocessor.constraint.NotNull;
+import org.supercsv.cellprocessor.constraint.RequireHashCode;
+import org.supercsv.cellprocessor.constraint.RequireSubStr;
+import org.supercsv.cellprocessor.constraint.StrMinMax;
+import org.supercsv.cellprocessor.constraint.StrNotNullOrEmpty;
+import org.supercsv.cellprocessor.constraint.StrRegEx;
+import org.supercsv.cellprocessor.constraint.Strlen;
+import org.supercsv.cellprocessor.constraint.Unique;
+import org.supercsv.cellprocessor.constraint.UniqueHashCode;
+import org.supercsv.cellprocessor.ift.CellProcessor;
+import org.supercsv.exception.SuperCsvCellProcessorException;
+import org.supercsv.io.CsvListReader;
+import org.supercsv.prefs.CsvPreference;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"csv", "schema", "validation"})
+@CapabilityDescription("Validates the contents of FlowFiles against a 
user-specified CSV schema. " +
+"Take a look at the additional documentation of this processor for 
some schema examples.")
+public class ValidateCsv extends AbstractProcessor {
+
+private final static List 

[jira] [Assigned] (NIFI-2411) ModifyBytes should use long instead of int for offsets.

[Joe Skora (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Skora reassigned NIFI-2411:
---

Assignee: Joe Skora

> ModifyBytes should use long instead of int for offsets.
> ---
>
> Key: NIFI-2411
> URL: https://issues.apache.org/jira/browse/NIFI-2411
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Joe Skora
>Assignee: Joe Skora
>  Labels: easyfix
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> ModifyBytes.onTrigger() uses Java 32 bit {{int}}s for byte offsets limiting 
> it to 2 Gigabytes, switching to {{long}}'s will all it to handle up to 15 
> Exabytes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-1867) improve ModifyBytes to make it easy to remove all flowfile content

[Joe Skora (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Skora reassigned NIFI-1867:
---

Assignee: Joe Skora

> improve ModifyBytes to make it easy to remove all flowfile content
> --
>
> Key: NIFI-1867
> URL: https://issues.apache.org/jira/browse/NIFI-1867
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.6.1
>Reporter: Ben Icore
>Assignee: Joe Skora
>
> update ModifyBytes processor to include a "Remove all content" property.  
> this property shouild default to false so existing functionality is not 
> changed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)