[jira] [Commented] (NIFI-2652) Handle multiple invocations of the encrypt-config tool

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15638630#comment-15638630
 ] 

ASF GitHub Bot commented on NIFI-2652:
--

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/1186

NIFI-2652 [WIP] Handle encrypted config key migration

Submitting a WIP PR because other features depend on this work. There is an 
unrelated test failure that I got locally after rebasing against master, so I 
will investigate that, but the module where I did all of this work is fine. 

I will also update the Admin Guide with instructions for key migration and 
squash these commits.  
---
Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [o] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [o] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-2652

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1186.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1186


commit ad2b8d46a21a1642f88be9519ab535d60930dfe5
Author: Andy LoPresto 
Date:   2016-11-02T03:49:15Z

NIFI-2652 Implemented first pass of key migration logic and provided single 
comprehensive unit test.

commit f7f88859e1b26e40d0abeb0670d2626ddf6f4714
Author: Andy LoPresto 
Date:   2016-11-03T02:31:37Z

NIFI-2652 Added logic and unit test for all combinations of original 
key/password and new key/password.

commit 283924dacaf6f7b4396334c4315d040da240eb71
Author: Andy LoPresto 
Date:   2016-11-05T04:08:02Z

NIFI-2652 Expanded unit test for combinations into individual tests due to 
System.exit() only be capturable once per test.
Three tests which mock Scrypt for speed are temporarily ignored to perform 
test pollution identification.

commit f28610a65e9245e9a8b2fe462efdcec8aa777331
Author: Andy LoPresto 
Date:   2016-11-05T04:11:48Z

NIFI-2652 Removed SCrypt mock from one unit test that did not need it. Two 
offenders remain ignored.

commit df9c18f4aae1f1dd40631f37139b91646bfa4e4f
Author: Andy LoPresto 
Date:   2016-11-05T04:14:13Z

NIFI-2652 Removed SCrypt mock from one redundant unit test. One offender 
remains ignored.

commit 23a150c52120f12a7f772dc655cf5d8e8223
Author: Andy LoPresto 
Date:   2016-11-05T04:22:51Z

NIFI-2652 Removed SCrypt mock from one unit test that didn't need it. Test 
pollution is removed and all tests pass.

commit f6144a491ad27f70cf562749e1b8da02364bb348
Author: Andy LoPresto 
Date:   2016-11-05T04:42:46Z

NIFI-2652 Added unit tests for negative cases for migration argument 
parsing.
Cleaned up TODOs and comments.




> Handle multiple invocations of the encrypt-config tool
> --
>
> Key: NIFI-2652
> URL: https://issues.apache.org/jira/browse/NIFI-2652
> Project: Apache NiFi
>  Issue Type: Improvement
>  

[GitHub] nifi pull request #1186: NIFI-2652 [WIP] Handle encrypted config key migrati...

[alopresto]

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/1186

NIFI-2652 [WIP] Handle encrypted config key migration

Submitting a WIP PR because other features depend on this work. There is an 
unrelated test failure that I got locally after rebasing against master, so I 
will investigate that, but the module where I did all of this work is fine. 

I will also update the Admin Guide with instructions for key migration and 
squash these commits.  
---
Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [o] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [o] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-2652

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1186.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1186


commit ad2b8d46a21a1642f88be9519ab535d60930dfe5
Author: Andy LoPresto 
Date:   2016-11-02T03:49:15Z

NIFI-2652 Implemented first pass of key migration logic and provided single 
comprehensive unit test.

commit f7f88859e1b26e40d0abeb0670d2626ddf6f4714
Author: Andy LoPresto 
Date:   2016-11-03T02:31:37Z

NIFI-2652 Added logic and unit test for all combinations of original 
key/password and new key/password.

commit 283924dacaf6f7b4396334c4315d040da240eb71
Author: Andy LoPresto 
Date:   2016-11-05T04:08:02Z

NIFI-2652 Expanded unit test for combinations into individual tests due to 
System.exit() only be capturable once per test.
Three tests which mock Scrypt for speed are temporarily ignored to perform 
test pollution identification.

commit f28610a65e9245e9a8b2fe462efdcec8aa777331
Author: Andy LoPresto 
Date:   2016-11-05T04:11:48Z

NIFI-2652 Removed SCrypt mock from one unit test that did not need it. Two 
offenders remain ignored.

commit df9c18f4aae1f1dd40631f37139b91646bfa4e4f
Author: Andy LoPresto 
Date:   2016-11-05T04:14:13Z

NIFI-2652 Removed SCrypt mock from one redundant unit test. One offender 
remains ignored.

commit 23a150c52120f12a7f772dc655cf5d8e8223
Author: Andy LoPresto 
Date:   2016-11-05T04:22:51Z

NIFI-2652 Removed SCrypt mock from one unit test that didn't need it. Test 
pollution is removed and all tests pass.

commit f6144a491ad27f70cf562749e1b8da02364bb348
Author: Andy LoPresto 
Date:   2016-11-05T04:42:46Z

NIFI-2652 Added unit tests for negative cases for migration argument 
parsing.
Cleaned up TODOs and comments.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-2992) Improve connection hover text by collecting all connection tooltips into one

[Andrew Lim (JIRA)]

Andrew Lim created NIFI-2992:


 Summary: Improve connection hover text by collecting all 
connection tooltips into one
 Key: NIFI-2992
 URL: https://issues.apache.org/jira/browse/NIFI-2992
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core UI
Affects Versions: 1.1.0
Reporter: Andrew Lim
Priority: Minor


With the addition of backpressure visual indicators and the flow expiration 
"clock" to connections, when the user hovers over different areas of the 
connection, different tooltips are displayed.

Instead of requiring the user to have the mouse in a specific/small area of the 
connection to get the tooltip with the desired information, change the tooltip 
to contain all of the information available for the connection.  So the "super" 
tooltip would include:

-FlowFile Expiration Progress/FlowFile Expiration Time Limit
-Back Pressure Object Threshold Queue %   (#Objects/Object Threshold)
-Back Pressure Data Size Threshold Queue %   (Data/Data Threshold)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi-minifi pull request #50: MINIFI-107 - Process group support

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/50#discussion_r86612327
  
--- Diff: 
minifi-toolkit/minifi-toolkit-configuration/src/test/resources/ProcessGroupsAndRemoteProcessGroups.yml
 ---
@@ -0,0 +1,276 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the \"License\"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an \"AS IS\" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 2
+Flow Controller:
+  name: ProcessGroupsAndRemoteProcessGroups
+  comment: ''
+Core Properties:
+  flow controller graceful shutdown period: 10 sec
+  flow service write delay interval: 500 ms
+  administrative yield duration: 30 sec
+  bored yield duration: 10 millis
+  max concurrent threads: 1
+FlowFile Repository:
+  partitions: 256
+  checkpoint interval: 2 mins
+  always sync: false
+  Swap:
+threshold: 2
+in period: 5 sec
+in threads: 1
+out period: 5 sec
+out threads: 4
+Content Repository:
+  content claim max appendable size: 10 MB
+  content claim max flow files: 100
+  always sync: false
+Provenance Repository:
+  provenance rollover time: 1 min
+Component Status Repository:
+  buffer size: 1440
+  snapshot frequency: 1 min
+Security Properties:
+  keystore: ''
+  keystore type: ''
+  keystore password: ''
+  key password: ''
+  truststore: ''
+  truststore type: ''
+  truststore password: ''
+  ssl protocol: ''
+  Sensitive Props:
+key:
+algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
+provider: BC
+Processors:
+- id: 207748d1-0158-1000--
+  name: GenerateFlowFile
+  class: org.apache.nifi.processors.standard.GenerateFlowFile
+  max concurrent tasks: 1
+  scheduling strategy: TIMER_DRIVEN
+  scheduling period: 0 sec
+  penalization period: 30 sec
+  yield period: 1 sec
+  run duration nanos: 0
+  auto-terminated relationships list: []
+  Properties:
+Batch Size: '1'
+Data Format: Binary
+File Size: 1 b
+Unique FlowFiles: 'false'
+- id: 2079e8bd-0158-1000--
+  name: LogAttribute
+  class: org.apache.nifi.processors.standard.LogAttribute
+  max concurrent tasks: 1
+  scheduling strategy: TIMER_DRIVEN
+  scheduling period: 0 sec
+  penalization period: 30 sec
+  yield period: 1 sec
+  run duration nanos: 0
+  auto-terminated relationships list:
+  - success
+  Properties:
+Attributes to Ignore:
+Attributes to Log:
+Log Level: info
+Log Payload: 'false'
+Log prefix:
+- id: 2077ab1e-0158-1000--
+  name: UpdateAttribute
+  class: org.apache.nifi.processors.attributes.UpdateAttribute
+  max concurrent tasks: 1
+  scheduling strategy: TIMER_DRIVEN
+  scheduling period: 0 sec
+  penalization period: 30 sec
+  yield period: 1 sec
+  run duration nanos: 0
+  auto-terminated relationships list: []
+  Properties:
+Delete Attributes Expression:
+top: top
+Process Groups:
+- id: 207888b1-0158-1000--
+  name: middle
+  Processors:
+  - id: 2078f34e-0158-1000--
+name: UpdateAttribute
+class: org.apache.nifi.processors.attributes.UpdateAttribute
+max concurrent tasks: 1
+scheduling strategy: TIMER_DRIVEN
+scheduling period: 0 sec
+penalization period: 30 sec
+yield period: 1 sec
+run duration nanos: 0
+auto-terminated relationships list: []
+Properties:
+  Delete Attributes Expression:
+  middle: middle
+  Process Groups:
+  - id: 20794cd4-0158-1000--
+name: bottom
+Processors:
+- id: 207a89ba-0158-1000--
+  name: UpdateAttribute
+  class: org.apache.nifi.processors.attributes.UpdateAttribute
+  max concurrent tasks: 1
+  scheduling strategy: TIMER_DRIVEN
+  scheduling period: 0 sec
+  penalization period: 30 sec
+  yield period: 

[GitHub] nifi-minifi pull request #50: MINIFI-107 - Process group support

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/50#discussion_r86615541
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/ConfigSchema.java
 ---
@@ -31,41 +33,33 @@
 import java.util.stream.Collectors;
 
 import static 
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.COMPONENT_STATUS_REPO_KEY;
-import static 
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.CONNECTIONS_KEY;
 import static 
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.CONTENT_REPO_KEY;
 import static 
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.CORE_PROPS_KEY;
 import static 
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.FLOWFILE_REPO_KEY;
 import static 
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.FLOW_CONTROLLER_PROPS_KEY;
-import static 
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.PROCESSORS_KEY;
 import static 
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.PROVENANCE_REPORTING_KEY;
 import static 
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.PROVENANCE_REPO_KEY;
-import static 
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.REMOTE_PROCESSING_GROUPS_KEY;
 import static 
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.SECURITY_PROPS_KEY;
 
-/**
- *
- */
 public class ConfigSchema extends BaseSchema implements WritableSchema, 
ConvertableSchema {
-public static final String FOUND_THE_FOLLOWING_DUPLICATE_PROCESSOR_IDS 
= "Found the following duplicate processor ids: ";
-public static final String 
FOUND_THE_FOLLOWING_DUPLICATE_CONNECTION_IDS = "Found the following duplicate 
connection ids: ";
-public static final String 
FOUND_THE_FOLLOWING_DUPLICATE_REMOTE_PROCESSING_GROUP_NAMES = "Found the 
following duplicate remote processing group names: ";
-public static final String 
FOUND_THE_FOLLOWING_DUPLICATE_REMOTE_INPUT_PORT_IDS = "Found the following 
duplicate remote input port ids: ";
-public static final String FOUND_THE_FOLLOWING_DUPLICATE_IDS = "Found 
the following ids that occur both in Processors and Remote Input Ports: ";
 public static final int CONFIG_VERSION = 2;
 public static final String VERSION = "MiNiFi Config Version";
+public static final String 
FOUND_THE_FOLLOWING_DUPLICATE_REMOTE_INPUT_PORT_IDS = "Found the following 
duplicate remote input port ids: ";
+public static final String FOUND_THE_FOLLOWING_DUPLICATE_IDS = "Found 
the following ids that occur both in Processors and Remote Input Ports: ";
--- End diff --

Where this message is used now covers all overlapping ids. This variable 
should be updated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #50: MINIFI-107 - Process group support

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/50#discussion_r86620366
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/RemoteProcessingGroupSchema.java
 ---
@@ -50,17 +48,18 @@
 
 public RemoteProcessingGroupSchema(Map map) {
 name = getRequiredKeyAsType(map, NAME_KEY, String.class, 
REMOTE_PROCESSING_GROUPS_KEY);
-url = getRequiredKeyAsType(map, URL_KEY, String.class, 
REMOTE_PROCESSING_GROUPS_KEY);
-inputPorts = convertListToType(getRequiredKeyAsType(map, 
INPUT_PORTS_KEY, List.class, REMOTE_PROCESSING_GROUPS_KEY), "input port", 
RemoteInputPortSchema.class, INPUT_PORTS_KEY);
+String wrapperName = "RemoteProcessingGroup(name: 
{name})".replace("{name}", StringUtil.isNullOrEmpty(name) ? "unknown" : name);
--- End diff --

This may be a basic question but why use ".replace" (which compiles a 
pattern) instead of just doing string concatenation?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #50: MINIFI-107 - Process group support

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/50#discussion_r86617837
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/ConfigSchema.java
 ---
@@ -101,47 +89,60 @@ public ConfigSchema(Map map, List 
validationIssues) {
 addIssuesIfNotNull(contentRepositoryProperties);
 addIssuesIfNotNull(componentStatusRepositoryProperties);
 addIssuesIfNotNull(securityProperties);
+addIssuesIfNotNull(processGroupSchema);
 addIssuesIfNotNull(provenanceReportingProperties);
 addIssuesIfNotNull(provenanceRepositorySchema);
-addIssuesIfNotNull(processors);
-addIssuesIfNotNull(connections);
-addIssuesIfNotNull(remoteProcessingGroups);
 
-Set processorIds = new HashSet<>();
-List processorIdList = 
processors.stream().map(ProcessorSchema::getId).collect(Collectors.toList());
-processorIds.addAll(processorIdList);
+List allProcessGroups = 
getAllProcessGroups(processGroupSchema);
+List allConnectionSchemas = 
allProcessGroups.stream().flatMap(p -> 
p.getConnections().stream()).collect(Collectors.toList());
+List allRemoteProcessingGroups = 
allProcessGroups.stream().flatMap(p -> 
p.getRemoteProcessingGroups().stream()).collect(Collectors.toList());
 
-checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_PROCESSOR_IDS, processorIdList);
-checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_CONNECTION_IDS, 
connections.stream().map(ConnectionSchema::getId).collect(Collectors.toList()));
-checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_REMOTE_PROCESSING_GROUP_NAMES,
-
remoteProcessingGroups.stream().map(RemoteProcessingGroupSchema::getName).collect(Collectors.toList()));
-
-Set remoteInputPortIds = new HashSet<>();
-List remoteInputPortIdList = 
remoteProcessingGroups.stream().filter(r -> r.getInputPorts() != null)
+List allProcessorIds = allProcessGroups.stream().flatMap(p 
-> 
p.getProcessors().stream()).map(ProcessorSchema::getId).collect(Collectors.toList());
+List allConnectionIds = 
allConnectionSchemas.stream().map(ConnectionSchema::getId).collect(Collectors.toList());
+List allRemoteProcessingGroupNames = 
allRemoteProcessingGroups.stream().map(RemoteProcessingGroupSchema::getName).collect(Collectors.toList());
+List allRemoteInputPortIds = 
allRemoteProcessingGroups.stream().filter(r -> r.getInputPorts() != null)
 .flatMap(r -> 
r.getInputPorts().stream()).map(RemoteInputPortSchema::getId).collect(Collectors.toList());
-checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_REMOTE_INPUT_PORT_IDS, remoteInputPortIdList);
-remoteInputPortIds.addAll(remoteInputPortIdList);
+List allInputPortIds = allProcessGroups.stream().flatMap(p 
-> 
p.getInputPortSchemas().stream()).map(PortSchema::getId).collect(Collectors.toList());
+List allOutputPortIds = 
allProcessGroups.stream().flatMap(p -> 
p.getOutputPortSchemas().stream()).map(PortSchema::getId).collect(Collectors.toList());
+
+checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_PROCESSOR_IDS, allProcessorIds);
+checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_CONNECTION_IDS, allConnectionIds);
+checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_REMOTE_PROCESSING_GROUP_NAMES, 
allRemoteProcessingGroupNames);
+checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_REMOTE_INPUT_PORT_IDS, allRemoteInputPortIds);
--- End diff --

There doesn't appear to be a "checkForDuplicates" for output ports. Is that 
intended?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #50: MINIFI-107 - Process group support

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/50#discussion_r86610341
  
--- Diff: minifi-docs/src/main/markdown/System_Admin_Guide.md ---
@@ -391,6 +391,39 @@ Within the Processor Configuration section, there is 
the `Properties` subsection
 State File: ./conf/state/tail-file
 Initial Start Position: Beginning of File
 
+## Process Groups
+
+Process groups can be nested from the top level.  They can contain other 
process groups as well and can be used to logically group related operations.
+
+*Property*  | *Description*
+--- | -
+name| The name of what this process group 
will do.
+id  | The id of this process group.  This 
needs to be set to a unique filesystem-friendly value (regex: [A-Za-z0-9_-]+)
+Processors  | The processors contained in this 
Process Group. (Defined above)
+Remote Processing Groups| The remote processing groups 
contained in this Process Group. (Defined below)
+Connections | The connections contained in this 
Process Group. (Defined below)
+Input Ports | The input ports contained in this 
Process Group. (Defined below)
+Output Ports| The output ports contained in this 
Process Group. (Defined below)
+Process Groups  | The child Process Groups contained 
in this Process Group.
+
+## Input Ports
+
+These ports provide input to the Process Group they reside on.
+
+*Property*   | *Description*
+ | -
+name | The name of what this input port will do.
+id   | The id of this input port.  This needs to be 
set to a unique filesystem-friendly value (regex: [A-Za-z0-9_-]+)
+
+## Input Ports
--- End diff --

I believe this should be "Output Ports"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #50: MINIFI-107 - Process group support

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/50#discussion_r86619586
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/ConnectionSchema.java
 ---
@@ -50,21 +49,18 @@
 private String queuePrioritizerClass;
 
 public ConnectionSchema(Map map) {
-super(map, CONNECTIONS_KEY);
+super(map, "Connection(id: {id}, name: {name})");
 
+String wrapperName = getWrapperName();
 // In case of older version, these may not be available until 
after construction, validated in getValidationIssues()
-sourceId = getOptionalKeyAsType(map, SOURCE_ID_KEY, String.class, 
CONNECTIONS_KEY, "");
-destinationId = getOptionalKeyAsType(map, DESTINATION_ID_KEY, 
String.class, CONNECTIONS_KEY, "");
-
-sourceRelationshipNames = getOptionalKeyAsType(map, 
SOURCE_RELATIONSHIP_NAMES_KEY, List.class, CONNECTIONS_KEY, new ArrayList<>());
-if (sourceRelationshipNames.isEmpty()) {
-addValidationIssue("Expected at least one value in " + 
SOURCE_RELATIONSHIP_NAMES_KEY + " for " + CONNECTIONS_KEY + " " + getName());
-}
-
-maxWorkQueueSize = getOptionalKeyAsType(map, 
MAX_WORK_QUEUE_SIZE_KEY, Number.class, CONNECTIONS_KEY, 
DEFAULT_MAX_WORK_QUEUE_SIZE);
-maxWorkQueueDataSize = getOptionalKeyAsType(map, 
MAX_WORK_QUEUE_DATA_SIZE_KEY, String.class, CONNECTIONS_KEY, 
DEFAULT_MAX_QUEUE_DATA_SIZE);
-flowfileExpiration = getOptionalKeyAsType(map, 
FLOWFILE_EXPIRATION__KEY, String.class, CONNECTIONS_KEY, 
DEFAULT_FLOWFILE_EXPIRATION);
-queuePrioritizerClass = getOptionalKeyAsType(map, 
QUEUE_PRIORITIZER_CLASS_KEY, String.class, CONNECTIONS_KEY, "");
+sourceId = getOptionalKeyAsType(map, SOURCE_ID_KEY, String.class, 
wrapperName, "");
+destinationId = getOptionalKeyAsType(map, DESTINATION_ID_KEY, 
String.class, wrapperName, "");
+
+sourceRelationshipNames = getOptionalKeyAsType(map, 
SOURCE_RELATIONSHIP_NAMES_KEY, List.class, wrapperName, new ArrayList<>());
--- End diff --

Could you add a comment explaining that this could be empty if the source 
is a port?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #50: MINIFI-107 - Process group support

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/50#discussion_r86619023
  
--- Diff: 
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/ConfigSchema.java
 ---
@@ -101,47 +89,60 @@ public ConfigSchema(Map map, List 
validationIssues) {
 addIssuesIfNotNull(contentRepositoryProperties);
 addIssuesIfNotNull(componentStatusRepositoryProperties);
 addIssuesIfNotNull(securityProperties);
+addIssuesIfNotNull(processGroupSchema);
 addIssuesIfNotNull(provenanceReportingProperties);
 addIssuesIfNotNull(provenanceRepositorySchema);
-addIssuesIfNotNull(processors);
-addIssuesIfNotNull(connections);
-addIssuesIfNotNull(remoteProcessingGroups);
 
-Set processorIds = new HashSet<>();
-List processorIdList = 
processors.stream().map(ProcessorSchema::getId).collect(Collectors.toList());
-processorIds.addAll(processorIdList);
+List allProcessGroups = 
getAllProcessGroups(processGroupSchema);
+List allConnectionSchemas = 
allProcessGroups.stream().flatMap(p -> 
p.getConnections().stream()).collect(Collectors.toList());
+List allRemoteProcessingGroups = 
allProcessGroups.stream().flatMap(p -> 
p.getRemoteProcessingGroups().stream()).collect(Collectors.toList());
 
-checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_PROCESSOR_IDS, processorIdList);
-checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_CONNECTION_IDS, 
connections.stream().map(ConnectionSchema::getId).collect(Collectors.toList()));
-checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_REMOTE_PROCESSING_GROUP_NAMES,
-
remoteProcessingGroups.stream().map(RemoteProcessingGroupSchema::getName).collect(Collectors.toList()));
-
-Set remoteInputPortIds = new HashSet<>();
-List remoteInputPortIdList = 
remoteProcessingGroups.stream().filter(r -> r.getInputPorts() != null)
+List allProcessorIds = allProcessGroups.stream().flatMap(p 
-> 
p.getProcessors().stream()).map(ProcessorSchema::getId).collect(Collectors.toList());
+List allConnectionIds = 
allConnectionSchemas.stream().map(ConnectionSchema::getId).collect(Collectors.toList());
+List allRemoteProcessingGroupNames = 
allRemoteProcessingGroups.stream().map(RemoteProcessingGroupSchema::getName).collect(Collectors.toList());
+List allRemoteInputPortIds = 
allRemoteProcessingGroups.stream().filter(r -> r.getInputPorts() != null)
 .flatMap(r -> 
r.getInputPorts().stream()).map(RemoteInputPortSchema::getId).collect(Collectors.toList());
-checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_REMOTE_INPUT_PORT_IDS, remoteInputPortIdList);
-remoteInputPortIds.addAll(remoteInputPortIdList);
+List allInputPortIds = allProcessGroups.stream().flatMap(p 
-> 
p.getInputPortSchemas().stream()).map(PortSchema::getId).collect(Collectors.toList());
+List allOutputPortIds = 
allProcessGroups.stream().flatMap(p -> 
p.getOutputPortSchemas().stream()).map(PortSchema::getId).collect(Collectors.toList());
+
+checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_PROCESSOR_IDS, allProcessorIds);
+checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_CONNECTION_IDS, allConnectionIds);
+checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_REMOTE_PROCESSING_GROUP_NAMES, 
allRemoteProcessingGroupNames);
+checkForDuplicates(this::addValidationIssue, 
FOUND_THE_FOLLOWING_DUPLICATE_REMOTE_INPUT_PORT_IDS, allRemoteInputPortIds);
 
-Set duplicateIds = new HashSet<>(processorIds);
-duplicateIds.retainAll(remoteInputPortIds);
-if (duplicateIds.size() > 0) {
-addValidationIssue(FOUND_THE_FOLLOWING_DUPLICATE_IDS + 
duplicateIds.stream().sorted().collect(Collectors.joining(", ")));
+OverlapResults overlapResults = findOverlap(new 
HashSet<>(allProcessorIds), new HashSet<>(allRemoteInputPortIds), new 
HashSet<>(allInputPortIds), new HashSet<>(allOutputPortIds));
+if (overlapResults.duplicates.size() > 0) {
+addValidationIssue(FOUND_THE_FOLLOWING_DUPLICATE_IDS + 
overlapResults.duplicates.stream().sorted().collect(Collectors.joining(", ")));
 }
 
-Set connectableIds = new HashSet<>(processorIds);
-connectableIds.addAll(remoteInputPortIds);
-connections.forEach(c -> {
+allConnectionSchemas.forEach(c -> {
--- End diff --

This block checks if the connections go to any component in any process 

[GitHub] nifi-minifi pull request #50: MINIFI-107 - Process group support

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi-minifi/pull/50#discussion_r86611212
  
--- Diff: minifi-docs/src/main/markdown/System_Admin_Guide.md ---
@@ -391,6 +391,39 @@ Within the Processor Configuration section, there is 
the `Properties` subsection
 State File: ./conf/state/tail-file
 Initial Start Position: Beginning of File
 
+## Process Groups
+
+Process groups can be nested from the top level.  They can contain other 
process groups as well and can be used to logically group related operations.
+
+*Property*  | *Description*
+--- | -
+name| The name of what this process group 
will do.
+id  | The id of this process group.  This 
needs to be set to a unique filesystem-friendly value (regex: [A-Za-z0-9_-]+)
+Processors  | The processors contained in this 
Process Group. (Defined above)
+Remote Processing Groups| The remote processing groups 
contained in this Process Group. (Defined below)
+Connections | The connections contained in this 
Process Group. (Defined below)
+Input Ports | The input ports contained in this 
Process Group. (Defined below)
+Output Ports| The output ports contained in this 
Process Group. (Defined below)
+Process Groups  | The child Process Groups contained 
in this Process Group.
+
+## Input Ports
--- End diff --

Since in this PR we're just enabling internal Input/Output ports. This 
needs to be documented.

That said, adding functionality for external I/O ports would probably be 
easy as a follow-up ticket (additional properties in the config.yml, 
validation, etc.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2950) Allow literal interpretation of hex numbers as whole numbers in EL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637253#comment-15637253
 ] 

ASF GitHub Bot commented on NIFI-2950:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1161#discussion_r86605783
  
--- Diff: 
nifi-commons/nifi-expression-language/src/main/java/org/apache/nifi/attribute/expression/language/evaluation/functions/FromRadixEvaluator.java
 ---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.attribute.expression.language.evaluation.functions;
+
+import org.apache.nifi.attribute.expression.language.evaluation.Evaluator;
+import 
org.apache.nifi.attribute.expression.language.evaluation.QueryResult;
+import 
org.apache.nifi.attribute.expression.language.evaluation.WholeNumberEvaluator;
+import 
org.apache.nifi.attribute.expression.language.evaluation.WholeNumberQueryResult;
+
+import java.util.Map;
+
+public class FromRadixEvaluator extends WholeNumberEvaluator {
+
+private final Evaluator numberEvaluator;
+private final Evaluator radixEvaluator;
+
+public FromRadixEvaluator(final Evaluator subject, final 
Evaluator radixEvaluator) {
+this.numberEvaluator = subject;
+this.radixEvaluator = radixEvaluator;
+}
+
+@Override
+public QueryResult evaluate(final Map 
attributes) {
+final String result = 
numberEvaluator.evaluate(attributes).getValue();
--- End diff --

I highly doubt this is considered part of the Public API. 

Also passing null to the evaluate() methods is expected but since I never 
explicitly use attributes (other than passing it to the next evaluator), it is 
safe from NPE.


> Allow literal interpretation of hex numbers as whole numbers in EL
> --
>
> Key: NIFI-2950
> URL: https://issues.apache.org/jira/browse/NIFI-2950
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Joseph Percivall
>Priority: Minor
>
> Currently in EL you cannot pass hex numbers as a whole number and instead 
> only decimals support them. Whole numbers should support being passed in as 
> hex too.
> This was brought up by this SO[1] question.
> [1] 
> http://stackoverflow.com/questions/40265791/applying-string-manipulations-mathematical-operations-to-the-contents-of-a-flow/40267507#40267507



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1161: NIFI-2950 Adding support for whole number hex value...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1161#discussion_r86605783
  
--- Diff: 
nifi-commons/nifi-expression-language/src/main/java/org/apache/nifi/attribute/expression/language/evaluation/functions/FromRadixEvaluator.java
 ---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.attribute.expression.language.evaluation.functions;
+
+import org.apache.nifi.attribute.expression.language.evaluation.Evaluator;
+import 
org.apache.nifi.attribute.expression.language.evaluation.QueryResult;
+import 
org.apache.nifi.attribute.expression.language.evaluation.WholeNumberEvaluator;
+import 
org.apache.nifi.attribute.expression.language.evaluation.WholeNumberQueryResult;
+
+import java.util.Map;
+
+public class FromRadixEvaluator extends WholeNumberEvaluator {
+
+private final Evaluator numberEvaluator;
+private final Evaluator radixEvaluator;
+
+public FromRadixEvaluator(final Evaluator subject, final 
Evaluator radixEvaluator) {
+this.numberEvaluator = subject;
+this.radixEvaluator = radixEvaluator;
+}
+
+@Override
+public QueryResult evaluate(final Map 
attributes) {
+final String result = 
numberEvaluator.evaluate(attributes).getValue();
--- End diff --

I highly doubt this is considered part of the Public API. 

Also passing null to the evaluate() methods is expected but since I never 
explicitly use attributes (other than passing it to the next evaluator), it is 
safe from NPE.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-401) New Scheduling strategy (On primary node - CRON )

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636986#comment-15636986
 ] 

ASF GitHub Bot commented on NIFI-401:
-

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/1185

NIFI-401: Decouple Scheduling Strategy from Execution Node

NIFI-401:
- Decoupling  scheduling strategy from execution node.
- Ensuring existing configuration is retained and shown until the user 
explicits changes it.
- Retaining, but disabling, deprecated options.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-401

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1185.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1185


commit 23f9a12c558cace9689e98b36e1edab772fbdce2
Author: Brian Eugley 
Date:   2016-06-09T15:32:15Z

NIFI-401

commit 7f8d4ab811434ef480b92ecd89b1b622ea146fb0
Author: Matt Gilman 
Date:   2016-11-04T15:03:35Z

NIFI-401:
- Minor tweaks to PR #1117.
- Ensuring existing configuraiton is retained and shown until the user 
explicits changes it.
- Retaining, but disabling, deprecated options.




> New Scheduling strategy (On primary node - CRON )
> -
>
> Key: NIFI-401
> URL: https://issues.apache.org/jira/browse/NIFI-401
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Matthew Clarke
> Attachments: initial prototype .png
>
>
> Currently the only scheduling strategy supported when a processor is set to 
> use "On primary Node" is Timer Driven.   There should be a second option to 
> allow cron driven On primary Node scheduling strategy.  This would allow 
> users to more control over when a given primary node only processor runs. 
> This would prevent these processors from running when configuration changes 
> or instance restarts occur.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1185: NIFI-401: Decouple Scheduling Strategy from Executi...

[mcgilman]

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/1185

NIFI-401: Decouple Scheduling Strategy from Execution Node

NIFI-401:
- Decoupling  scheduling strategy from execution node.
- Ensuring existing configuration is retained and shown until the user 
explicits changes it.
- Retaining, but disabling, deprecated options.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-401

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1185.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1185


commit 23f9a12c558cace9689e98b36e1edab772fbdce2
Author: Brian Eugley 
Date:   2016-06-09T15:32:15Z

NIFI-401

commit 7f8d4ab811434ef480b92ecd89b1b622ea146fb0
Author: Matt Gilman 
Date:   2016-11-04T15:03:35Z

NIFI-401:
- Minor tweaks to PR #1117.
- Ensuring existing configuraiton is retained and shown until the user 
explicits changes it.
- Retaining, but disabling, deprecated options.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2948) AWS DynamoDB Processors Do Not Update Credentials

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636760#comment-15636760
 ] 

ASF GitHub Bot commented on NIFI-2948:
--

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/1182
  
Thank you, @olegz 


> AWS DynamoDB Processors Do Not Update Credentials
> -
>
> Key: NIFI-2948
> URL: https://issues.apache.org/jira/browse/NIFI-2948
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.1.0
>
>
> Following a report to the dev email list, there appears to be a bug in the 
> AWS DynamoDB processors where the AWS credentials are cached by the processor 
> once started, and not updated even if the processor is stopped, reconfigured, 
> and started.
> The problem appears to be 
> [AbstractDynamoDBProcessor::getDynamoDB|https://github.com/apache/nifi/blob/rel/nifi-1.0.0/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/dynamodb/AbstractDynamoDBProcessor.java#L218],
>  which assigns the dynamo client object to a class variable only once.
> *Repro Steps*
> # Add GetDynamoDB processor
> # Configure GetDynamoDB required properties but *not* any credential fields
> # Start the processor
> # Run a flowfile through - it should fail with 
> "MissingAuthenticationTokenException" error
> # Stop the GetDynamoDB processor
> # Configure valid AWS credentials
> # Start the processor
> # Run a flowfile through - it should fail again with 
> "MissingAuthenticationTokenException" still using the old credentials



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1182: NIFI-2948 Fix DynamoDB Cached Credentials

[jvwing]

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/1182
  
Thank you, @olegz 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1184: NIFI-1002: Added WebSocket support.

[ijokarumawak]

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/1184
  
Hello NiFi developers,

This PR contains a new set of WebSocket support components.

At least there're two WebSocket processors available on GitHub already, 
[nifi-websocket](https://github.com/xmlking/nifi-websocket) by @xmlking , and 
[nifi-websockets-bundle](https://github.com/acesir/nifi-websockets-bundle) by 
@acesir. I appreciate these projects providing WebSocket support for NiFi, and 
wanted to merge the capability to Apache NiFi code base. I referred these 
existing projects, and add few functionalities and Unit Test codes.

Features:
- As a WebSocket client to connect a remote WebSocket server
- As a WebSocket server to receive connection from remote WebSocket clients
- Receive and send text messages as well as binary messages
- Secure communication (using SslContextService) is supported

It may look more complex, so for the description purpose, I wrote a blog 
post about how it works, and how it's designed. Please check this post as well, 
[NiFi WebSocket 
Support](http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/).

Any comment or advice would be appreciated, thanks in advance!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1002) support for Listen WebSocket processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636600#comment-15636600
 ] 

ASF GitHub Bot commented on NIFI-1002:
--

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/1184

NIFI-1002: Added WebSocket support.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [x] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-1002

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1184.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1184


commit 1c3f2baa015d53326410cb0b5c5fa1973222aa30
Author: Koji Kawamura 
Date:   2016-10-18T01:43:42Z

NIFI-1002: Added WebSocket support.




> support for Listen WebSocket processor 
> ---
>
> Key: NIFI-1002
> URL: https://issues.apache.org/jira/browse/NIFI-1002
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.4.0
>Reporter: sumanth chinthagunta
>Priority: Minor
>  Labels: newbie
>
> A WebSocket listen processor will be helpful for IoT data ingestion.
> I am playing with embedded Vert.X  for WebSocket and also ability to  put 
> FlowFiles back to WebSocket client via Vert.X EventBus.
> https://github.com/xmlking/nifi-websocket 
> I am new to NiFi. any advise can be  helpful.  
> PS: I feel forcing Interfaces for Controller Services is unnecessary as in 
> many cases Controller Services are only used by a set of Processors and 
> developers usually  bundle them together. 
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1184: NIFI-1002: Added WebSocket support.

[ijokarumawak]

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/1184

NIFI-1002: Added WebSocket support.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [x] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-1002

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1184.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1184


commit 1c3f2baa015d53326410cb0b5c5fa1973222aa30
Author: Koji Kawamura 
Date:   2016-10-18T01:43:42Z

NIFI-1002: Added WebSocket support.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2989) UI - Improve Update Policy dialog (Title, Text and Buttons)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636495#comment-15636495
 ] 

ASF GitHub Bot commented on NIFI-2989:
--

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/1183

UI - Updating the text and header of the delete policy dialog

NIFI-2989:
- Updating the content of the delete policy dialog to make it a little more 
clear that the permissions for the component will revert back to the inherit 
policy.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-2989

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1183.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1183


commit 576f1f8dd3e97b14a59b14cbf5ec72de0208bdf8
Author: Matt Gilman 
Date:   2016-11-04T14:21:52Z

NIFI-2989:
- Updating the content of the delete policy dialog to make it a little more 
clear that the permissions for the component will revert back to the inherit 
policy.




> UI - Improve Update Policy dialog (Title, Text and Buttons)
> ---
>
> Key: NIFI-2989
> URL: https://issues.apache.org/jira/browse/NIFI-2989
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Andrew Lim
>Assignee: Matt Gilman
>Priority: Minor
> Fix For: 1.1.0
>
> Attachments: NIFI-2989_UpdatePolicyDialog.png
>
>
> Referencing the attached screenshot:
> 1.  Change the title "Update Policy" to "Delete Policy"
> 2.  After the text "Are you sure you want to delete this policy?" add another 
> paragraph with:
> "By deleting this policy, the inherited policy will be reinstated."
> 3. Change the buttons from "Yes/No" to "Cancel/Delete".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1183: UI - Updating the text and header of the delete pol...

[mcgilman]

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/1183

UI - Updating the text and header of the delete policy dialog

NIFI-2989:
- Updating the content of the delete policy dialog to make it a little more 
clear that the permissions for the component will revert back to the inherit 
policy.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-2989

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1183.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1183


commit 576f1f8dd3e97b14a59b14cbf5ec72de0208bdf8
Author: Matt Gilman 
Date:   2016-11-04T14:21:52Z

NIFI-2989:
- Updating the content of the delete policy dialog to make it a little more 
clear that the permissions for the component will revert back to the inherit 
policy.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2989) UI - Improve Update Policy dialog (Title, Text and Buttons)

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-2989:
--
Fix Version/s: 1.1.0

> UI - Improve Update Policy dialog (Title, Text and Buttons)
> ---
>
> Key: NIFI-2989
> URL: https://issues.apache.org/jira/browse/NIFI-2989
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Andrew Lim
>Assignee: Matt Gilman
>Priority: Minor
> Fix For: 1.1.0
>
> Attachments: NIFI-2989_UpdatePolicyDialog.png
>
>
> Referencing the attached screenshot:
> 1.  Change the title "Update Policy" to "Delete Policy"
> 2.  After the text "Are you sure you want to delete this policy?" add another 
> paragraph with:
> "By deleting this policy, the inherited policy will be reinstated."
> 3. Change the buttons from "Yes/No" to "Cancel/Delete".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (NIFI-1322) Support appending files in PutHDFS

[Oleg Zhurakousky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky resolved NIFI-1322.

Resolution: Fixed

> Support appending files in PutHDFS
> --
>
> Key: NIFI-1322
> URL: https://issues.apache.org/jira/browse/NIFI-1322
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Scott
>Assignee: Pierre Villard
> Fix For: 1.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1322) Support appending files in PutHDFS

[Oleg Zhurakousky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky updated NIFI-1322:
---
Fix Version/s: 1.1.0

> Support appending files in PutHDFS
> --
>
> Key: NIFI-1322
> URL: https://issues.apache.org/jira/browse/NIFI-1322
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Scott
>Assignee: Pierre Villard
> Fix For: 1.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1322) Support appending files in PutHDFS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636199#comment-15636199
 ] 

ASF GitHub Bot commented on NIFI-1322:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1181


> Support appending files in PutHDFS
> --
>
> Key: NIFI-1322
> URL: https://issues.apache.org/jira/browse/NIFI-1322
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Scott
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1322) Support appending files in PutHDFS

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636196#comment-15636196
 ] 

ASF subversion and git services commented on NIFI-1322:
---

Commit a7d06412f8e4809157bf1074d659b5420546190f in nifi's branch 
refs/heads/master from [~pvillard]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=a7d0641 ]

NIFI-1322 - PutHDFS - allow file append resolution


> Support appending files in PutHDFS
> --
>
> Key: NIFI-1322
> URL: https://issues.apache.org/jira/browse/NIFI-1322
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Scott
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1322) Support appending files in PutHDFS

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636197#comment-15636197
 ] 

ASF subversion and git services commented on NIFI-1322:
---

Commit c13cfa6ea6009070db74d7dd9be9c66703d56942 in nifi's branch 
refs/heads/master from [~ozhurakousky]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=c13cfa6 ]

NIFI-1322 fixed breaking changes introduced in previous commit

This closes #1181


> Support appending files in PutHDFS
> --
>
> Key: NIFI-1322
> URL: https://issues.apache.org/jira/browse/NIFI-1322
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Scott
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1322) Support appending files in PutHDFS

[Oleg Zhurakousky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky updated NIFI-1322:
---
Assignee: Pierre Villard

> Support appending files in PutHDFS
> --
>
> Key: NIFI-1322
> URL: https://issues.apache.org/jira/browse/NIFI-1322
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Scott
>Assignee: Pierre Villard
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1181: NIFI-1322 - PutHDFS - allow file append resolution

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1181


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2909) Provide a framework mechanism for loading additional classpath resources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636116#comment-15636116
 ] 

ASF GitHub Bot commented on NIFI-2909:
--

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/1156
  
@bbende just left a few minor comments which probably all fall in the 
category of stylistic, so it's up to you to decide if you want to change 
anything. 
Now I need to spend some time and play with it to see exactly how it works. 
Will do it over the weekend.


> Provide a framework mechanism for loading additional classpath resources
> 
>
> Key: NIFI-2909
> URL: https://issues.apache.org/jira/browse/NIFI-2909
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
> Fix For: 1.1.0
>
>
> We currently have several components with a property for specifying 
> additional classpath resources (DBCP connection pool, scripting processors, 
> JMS). Each of these components is responsible for handling this in its own 
> way. 
> The framework should provide a more integrated solution to make it easier for 
> component developers to deal with this scenario. Some requirements that need 
> to be met by this solution:
> - Multiple instances of the same component with different resources added to 
> the classpath and not interfering with each other (i.e. two DBCP connection 
> pools using different drivers)
> - Ability to modify the actual ClassLoader of the component to deal with 
> frameworks that use Class.forName() without passing in a ClassLoader, meaning 
> if a processor loads class A and class A calls Class.forName(classBName), 
> then class B needs to be available in the ClassLoader that loaded the 
> processor's class which in turn loaded class A
> - A component developer should be able to indicate that a given 
> PropertyDescriptor represents a classpath resource and the framework should 
> take care of the ClassLoader manipulation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1156: NIFI-2909 and NIFI-1712 Per-Instance ClassLoading

[olegz]

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/1156
  
@bbende just left a few minor comments which probably all fall in the 
category of stylistic, so it's up to you to decide if you want to change 
anything. 
Now I need to spend some time and play with it to see exactly how it works. 
Will do it over the weekend.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1322) Support appending files in PutHDFS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636100#comment-15636100
 ] 

ASF GitHub Bot commented on NIFI-1322:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1181#discussion_r86528178
  
--- Diff: 
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
 ---
@@ -315,21 +328,24 @@ public void process(InputStream in) throws 
IOException {
 final long millis = 
stopWatch.getDuration(TimeUnit.MILLISECONDS);
 tempDotCopyFile = tempCopyFile;
 
-boolean renamed = false;
-for (int i = 0; i < 10; i++) { // try to rename multiple times.
-if (hdfs.rename(tempCopyFile, copyFile)) {
-renamed = true;
-break;// rename was successful
+if(!conflictResponse.equals(APPEND_RESOLUTION.getValue())
+|| 
(conflictResponse.equals(APPEND_RESOLUTION.getValue()) && !destinationExists)) {
+boolean renamed = false;
+for (int i = 0; i < 10; i++) { // try to rename multiple 
times.
+if (hdfs.rename(tempCopyFile, copyFile)) {
+renamed = true;
+break;// rename was successful
+}
+Thread.sleep(200L);// try waiting to let whatever 
might cause rename failure to resolve
--- End diff --

To be honest, no I don't. I assume that there is some kind of asynchronous 
call we are doing before when talking to remote clusters and that we might need 
to make this try multiple times to actually succeed. Not sure if it is still 
required with the last versions of HDFS client but didn't want to change it.


> Support appending files in PutHDFS
> --
>
> Key: NIFI-1322
> URL: https://issues.apache.org/jira/browse/NIFI-1322
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Scott
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1181: NIFI-1322 - PutHDFS - allow file append resolution

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1181#discussion_r86528178
  
--- Diff: 
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
 ---
@@ -315,21 +328,24 @@ public void process(InputStream in) throws 
IOException {
 final long millis = 
stopWatch.getDuration(TimeUnit.MILLISECONDS);
 tempDotCopyFile = tempCopyFile;
 
-boolean renamed = false;
-for (int i = 0; i < 10; i++) { // try to rename multiple times.
-if (hdfs.rename(tempCopyFile, copyFile)) {
-renamed = true;
-break;// rename was successful
+if(!conflictResponse.equals(APPEND_RESOLUTION.getValue())
+|| 
(conflictResponse.equals(APPEND_RESOLUTION.getValue()) && !destinationExists)) {
+boolean renamed = false;
+for (int i = 0; i < 10; i++) { // try to rename multiple 
times.
+if (hdfs.rename(tempCopyFile, copyFile)) {
+renamed = true;
+break;// rename was successful
+}
+Thread.sleep(200L);// try waiting to let whatever 
might cause rename failure to resolve
--- End diff --

To be honest, no I don't. I assume that there is some kind of asynchronous 
call we are doing before when talking to remote clusters and that we might need 
to make this try multiple times to actually succeed. Not sure if it is still 
required with the last versions of HDFS client but didn't want to change it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2909) Provide a framework mechanism for loading additional classpath resources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636084#comment-15636084
 ] 

ASF GitHub Bot commented on NIFI-2909:
--

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1156#discussion_r86527582
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractConfiguredComponent.java
 ---
@@ -141,48 +181,74 @@ public void setProperty(final String name, final 
String value) {
  * @return true if removed; false otherwise
  * @throws java.lang.IllegalArgumentException if the name is null
  */
-@Override
-public boolean removeProperty(final String name) {
+private boolean removeProperty(final String name) {
 if (null == name) {
 throw new IllegalArgumentException();
 }
 
-lock.lock();
-try {
-verifyModifiable();
+final PropertyDescriptor descriptor = 
component.getPropertyDescriptor(name);
+String value = null;
+if (!descriptor.isRequired() && (value = 
properties.remove(descriptor)) != null) {
 
-try (final NarCloseable narCloseable = 
NarCloseable.withComponentNarLoader(component.getClass())) {
-final PropertyDescriptor descriptor = 
component.getPropertyDescriptor(name);
-String value = null;
-if (!descriptor.isRequired() && (value = 
properties.remove(descriptor)) != null) {
-
-if (descriptor.getControllerServiceDefinition() != 
null) {
-if (value != null) {
-final ControllerServiceNode oldNode = 
serviceProvider.getControllerServiceNode(value);
-if (oldNode != null) {
-oldNode.removeReference(this);
-}
-}
+if (descriptor.getControllerServiceDefinition() != null) {
+if (value != null) {
+final ControllerServiceNode oldNode = 
serviceProvider.getControllerServiceNode(value);
+if (oldNode != null) {
+oldNode.removeReference(this);
 }
+}
+}
 
-try {
-component.onPropertyModified(descriptor, value, 
null);
-} catch (final Exception e) {
-// nothing really to do here...
-}
+try {
+component.onPropertyModified(descriptor, value, null);
+} catch (final Exception e) {
+// nothing really to do here...
--- End diff --

Same as above, just doesn't look right


> Provide a framework mechanism for loading additional classpath resources
> 
>
> Key: NIFI-2909
> URL: https://issues.apache.org/jira/browse/NIFI-2909
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
> Fix For: 1.1.0
>
>
> We currently have several components with a property for specifying 
> additional classpath resources (DBCP connection pool, scripting processors, 
> JMS). Each of these components is responsible for handling this in its own 
> way. 
> The framework should provide a more integrated solution to make it easier for 
> component developers to deal with this scenario. Some requirements that need 
> to be met by this solution:
> - Multiple instances of the same component with different resources added to 
> the classpath and not interfering with each other (i.e. two DBCP connection 
> pools using different drivers)
> - Ability to modify the actual ClassLoader of the component to deal with 
> frameworks that use Class.forName() without passing in a ClassLoader, meaning 
> if a processor loads class A and class A calls Class.forName(classBName), 
> then class B needs to be available in the ClassLoader that loaded the 
> processor's class which in turn loaded class A
> - A component developer should be able to indicate that a given 
> PropertyDescriptor represents a classpath resource and the framework should 
> take care of the ClassLoader manipulation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2909) Provide a framework mechanism for loading additional classpath resources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636071#comment-15636071
 ] 

ASF GitHub Bot commented on NIFI-2909:
--

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1156#discussion_r86527362
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractConfiguredComponent.java
 ---
@@ -90,47 +107,80 @@ public void setAnnotationData(final String data) {
 }
 
 @Override
-public void setProperty(final String name, final String value) {
-if (null == name || null == value) {
-throw new IllegalArgumentException();
+public void setProperties(Map properties) {
+if (properties == null) {
+return;
 }
 
 lock.lock();
 try {
 verifyModifiable();
 
-try (final NarCloseable narCloseable = 
NarCloseable.withComponentNarLoader(component.getClass())) {
-final PropertyDescriptor descriptor = 
component.getPropertyDescriptor(name);
-
-final String oldValue = properties.put(descriptor, value);
-if (!value.equals(oldValue)) {
-
-if (descriptor.getControllerServiceDefinition() != 
null) {
-if (oldValue != null) {
-final ControllerServiceNode oldNode = 
serviceProvider.getControllerServiceNode(oldValue);
-if (oldNode != null) {
-oldNode.removeReference(this);
-}
-}
-
-final ControllerServiceNode newNode = 
serviceProvider.getControllerServiceNode(value);
-if (newNode != null) {
-newNode.addReference(this);
+try (final NarCloseable narCloseable = 
NarCloseable.withComponentNarLoader(component.getClass(), id)) {
+final Set modulePaths = new LinkedHashSet<>();
+for (final Map.Entry entry : 
properties.entrySet()) {
+if (entry.getKey() != null && entry.getValue() == 
null) {
+removeProperty(entry.getKey());
+} else if (entry.getKey() != null) {
+setProperty(entry.getKey(), entry.getValue());
+
+// for any properties that dynamically modify the 
classpath, attempt to evaluate them for expression language
+final PropertyDescriptor descriptor = 
component.getPropertyDescriptor(entry.getKey());
+if (descriptor.isDynamicClasspathModifier() && 
!StringUtils.isEmpty(entry.getValue())) {
+final StandardPropertyValue propertyValue = 
new StandardPropertyValue(entry.getValue(), null, variableRegistry);
+
modulePaths.add(propertyValue.evaluateAttributeExpressions().getValue());
 }
 }
+}
 
-try {
-component.onPropertyModified(descriptor, oldValue, 
value);
-} catch (final Exception e) {
-// nothing really to do here...
-}
+final boolean requiresInstanceClassLoading = 
ExtensionManager.requiresInstanceClassLoading(component.getClass().getTypeName());
+if (requiresInstanceClassLoading) {
+processClasspathModifiers(modulePaths);
+} else if (!requiresInstanceClassLoading && 
modulePaths.size() > 0) {
+// Component has property descriptors with 
dynamicallyModifiesClasspath set to true, but the class does not have 
@RequiresInstanceClassLoading
+logger.warn("One or more property descriptors intend 
to modify the classpath, " +
+"but component does not contain the {} 
annotation, classpath will not be modified",
+new Object[] 
{RequiresInstanceClassLoading.class.getName()});
 }
 }
 } finally {
 lock.unlock();
 }
 }
 
+// Keep setProperty/removeProperty private so that all calls go 
through setProperties
+private void setProperty(final String name, final String value) {
+if (null == name || null == value) {
+throw new IllegalArgumentException();
+}
+
+final PropertyDescriptor 

[GitHub] nifi pull request #1156: NIFI-2909 and NIFI-1712 Per-Instance ClassLoading

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1156#discussion_r86527582
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractConfiguredComponent.java
 ---
@@ -141,48 +181,74 @@ public void setProperty(final String name, final 
String value) {
  * @return true if removed; false otherwise
  * @throws java.lang.IllegalArgumentException if the name is null
  */
-@Override
-public boolean removeProperty(final String name) {
+private boolean removeProperty(final String name) {
 if (null == name) {
 throw new IllegalArgumentException();
 }
 
-lock.lock();
-try {
-verifyModifiable();
+final PropertyDescriptor descriptor = 
component.getPropertyDescriptor(name);
+String value = null;
+if (!descriptor.isRequired() && (value = 
properties.remove(descriptor)) != null) {
 
-try (final NarCloseable narCloseable = 
NarCloseable.withComponentNarLoader(component.getClass())) {
-final PropertyDescriptor descriptor = 
component.getPropertyDescriptor(name);
-String value = null;
-if (!descriptor.isRequired() && (value = 
properties.remove(descriptor)) != null) {
-
-if (descriptor.getControllerServiceDefinition() != 
null) {
-if (value != null) {
-final ControllerServiceNode oldNode = 
serviceProvider.getControllerServiceNode(value);
-if (oldNode != null) {
-oldNode.removeReference(this);
-}
-}
+if (descriptor.getControllerServiceDefinition() != null) {
+if (value != null) {
+final ControllerServiceNode oldNode = 
serviceProvider.getControllerServiceNode(value);
+if (oldNode != null) {
+oldNode.removeReference(this);
 }
+}
+}
 
-try {
-component.onPropertyModified(descriptor, value, 
null);
-} catch (final Exception e) {
-// nothing really to do here...
-}
+try {
+component.onPropertyModified(descriptor, value, null);
+} catch (final Exception e) {
+// nothing really to do here...
--- End diff --

Same as above, just doesn't look right


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1156: NIFI-2909 and NIFI-1712 Per-Instance ClassLoading

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1156#discussion_r86527362
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractConfiguredComponent.java
 ---
@@ -90,47 +107,80 @@ public void setAnnotationData(final String data) {
 }
 
 @Override
-public void setProperty(final String name, final String value) {
-if (null == name || null == value) {
-throw new IllegalArgumentException();
+public void setProperties(Map properties) {
+if (properties == null) {
+return;
 }
 
 lock.lock();
 try {
 verifyModifiable();
 
-try (final NarCloseable narCloseable = 
NarCloseable.withComponentNarLoader(component.getClass())) {
-final PropertyDescriptor descriptor = 
component.getPropertyDescriptor(name);
-
-final String oldValue = properties.put(descriptor, value);
-if (!value.equals(oldValue)) {
-
-if (descriptor.getControllerServiceDefinition() != 
null) {
-if (oldValue != null) {
-final ControllerServiceNode oldNode = 
serviceProvider.getControllerServiceNode(oldValue);
-if (oldNode != null) {
-oldNode.removeReference(this);
-}
-}
-
-final ControllerServiceNode newNode = 
serviceProvider.getControllerServiceNode(value);
-if (newNode != null) {
-newNode.addReference(this);
+try (final NarCloseable narCloseable = 
NarCloseable.withComponentNarLoader(component.getClass(), id)) {
+final Set modulePaths = new LinkedHashSet<>();
+for (final Map.Entry entry : 
properties.entrySet()) {
+if (entry.getKey() != null && entry.getValue() == 
null) {
+removeProperty(entry.getKey());
+} else if (entry.getKey() != null) {
+setProperty(entry.getKey(), entry.getValue());
+
+// for any properties that dynamically modify the 
classpath, attempt to evaluate them for expression language
+final PropertyDescriptor descriptor = 
component.getPropertyDescriptor(entry.getKey());
+if (descriptor.isDynamicClasspathModifier() && 
!StringUtils.isEmpty(entry.getValue())) {
+final StandardPropertyValue propertyValue = 
new StandardPropertyValue(entry.getValue(), null, variableRegistry);
+
modulePaths.add(propertyValue.evaluateAttributeExpressions().getValue());
 }
 }
+}
 
-try {
-component.onPropertyModified(descriptor, oldValue, 
value);
-} catch (final Exception e) {
-// nothing really to do here...
-}
+final boolean requiresInstanceClassLoading = 
ExtensionManager.requiresInstanceClassLoading(component.getClass().getTypeName());
+if (requiresInstanceClassLoading) {
+processClasspathModifiers(modulePaths);
+} else if (!requiresInstanceClassLoading && 
modulePaths.size() > 0) {
+// Component has property descriptors with 
dynamicallyModifiesClasspath set to true, but the class does not have 
@RequiresInstanceClassLoading
+logger.warn("One or more property descriptors intend 
to modify the classpath, " +
+"but component does not contain the {} 
annotation, classpath will not be modified",
+new Object[] 
{RequiresInstanceClassLoading.class.getName()});
 }
 }
 } finally {
 lock.unlock();
 }
 }
 
+// Keep setProperty/removeProperty private so that all calls go 
through setProperties
+private void setProperty(final String name, final String value) {
+if (null == name || null == value) {
+throw new IllegalArgumentException();
+}
+
+final PropertyDescriptor descriptor = 
component.getPropertyDescriptor(name);
+
+final String oldValue = properties.put(descriptor, value);
+if (!value.equals(oldValue)) {
+
+if 

[jira] [Commented] (NIFI-2909) Provide a framework mechanism for loading additional classpath resources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636068#comment-15636068
 ] 

ASF GitHub Bot commented on NIFI-2909:
--

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1156#discussion_r86527207
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractConfiguredComponent.java
 ---
@@ -90,47 +107,80 @@ public void setAnnotationData(final String data) {
 }
 
 @Override
-public void setProperty(final String name, final String value) {
-if (null == name || null == value) {
-throw new IllegalArgumentException();
+public void setProperties(Map properties) {
+if (properties == null) {
+return;
 }
 
 lock.lock();
 try {
 verifyModifiable();
 
-try (final NarCloseable narCloseable = 
NarCloseable.withComponentNarLoader(component.getClass())) {
-final PropertyDescriptor descriptor = 
component.getPropertyDescriptor(name);
-
-final String oldValue = properties.put(descriptor, value);
-if (!value.equals(oldValue)) {
-
-if (descriptor.getControllerServiceDefinition() != 
null) {
-if (oldValue != null) {
-final ControllerServiceNode oldNode = 
serviceProvider.getControllerServiceNode(oldValue);
-if (oldNode != null) {
-oldNode.removeReference(this);
-}
-}
-
-final ControllerServiceNode newNode = 
serviceProvider.getControllerServiceNode(value);
-if (newNode != null) {
-newNode.addReference(this);
+try (final NarCloseable narCloseable = 
NarCloseable.withComponentNarLoader(component.getClass(), id)) {
+final Set modulePaths = new LinkedHashSet<>();
+for (final Map.Entry entry : 
properties.entrySet()) {
+if (entry.getKey() != null && entry.getValue() == 
null) {
+removeProperty(entry.getKey());
+} else if (entry.getKey() != null) {
+setProperty(entry.getKey(), entry.getValue());
+
+// for any properties that dynamically modify the 
classpath, attempt to evaluate them for expression language
+final PropertyDescriptor descriptor = 
component.getPropertyDescriptor(entry.getKey());
+if (descriptor.isDynamicClasspathModifier() && 
!StringUtils.isEmpty(entry.getValue())) {
+final StandardPropertyValue propertyValue = 
new StandardPropertyValue(entry.getValue(), null, variableRegistry);
+
modulePaths.add(propertyValue.evaluateAttributeExpressions().getValue());
 }
 }
+}
 
-try {
-component.onPropertyModified(descriptor, oldValue, 
value);
-} catch (final Exception e) {
-// nothing really to do here...
-}
+final boolean requiresInstanceClassLoading = 
ExtensionManager.requiresInstanceClassLoading(component.getClass().getTypeName());
+if (requiresInstanceClassLoading) {
+processClasspathModifiers(modulePaths);
+} else if (!requiresInstanceClassLoading && 
modulePaths.size() > 0) {
+// Component has property descriptors with 
dynamicallyModifiesClasspath set to true, but the class does not have 
@RequiresInstanceClassLoading
+logger.warn("One or more property descriptors intend 
to modify the classpath, " +
+"but component does not contain the {} 
annotation, classpath will not be modified",
+new Object[] 
{RequiresInstanceClassLoading.class.getName()});
 }
 }
 } finally {
 lock.unlock();
 }
 }
 
+// Keep setProperty/removeProperty private so that all calls go 
through setProperties
+private void setProperty(final String name, final String value) {
+if (null == name || null == value) {
+throw new IllegalArgumentException();
+}
--- End diff --

I see, but I still 

[GitHub] nifi pull request #1156: NIFI-2909 and NIFI-1712 Per-Instance ClassLoading

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1156#discussion_r86527207
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractConfiguredComponent.java
 ---
@@ -90,47 +107,80 @@ public void setAnnotationData(final String data) {
 }
 
 @Override
-public void setProperty(final String name, final String value) {
-if (null == name || null == value) {
-throw new IllegalArgumentException();
+public void setProperties(Map properties) {
+if (properties == null) {
+return;
 }
 
 lock.lock();
 try {
 verifyModifiable();
 
-try (final NarCloseable narCloseable = 
NarCloseable.withComponentNarLoader(component.getClass())) {
-final PropertyDescriptor descriptor = 
component.getPropertyDescriptor(name);
-
-final String oldValue = properties.put(descriptor, value);
-if (!value.equals(oldValue)) {
-
-if (descriptor.getControllerServiceDefinition() != 
null) {
-if (oldValue != null) {
-final ControllerServiceNode oldNode = 
serviceProvider.getControllerServiceNode(oldValue);
-if (oldNode != null) {
-oldNode.removeReference(this);
-}
-}
-
-final ControllerServiceNode newNode = 
serviceProvider.getControllerServiceNode(value);
-if (newNode != null) {
-newNode.addReference(this);
+try (final NarCloseable narCloseable = 
NarCloseable.withComponentNarLoader(component.getClass(), id)) {
+final Set modulePaths = new LinkedHashSet<>();
+for (final Map.Entry entry : 
properties.entrySet()) {
+if (entry.getKey() != null && entry.getValue() == 
null) {
+removeProperty(entry.getKey());
+} else if (entry.getKey() != null) {
+setProperty(entry.getKey(), entry.getValue());
+
+// for any properties that dynamically modify the 
classpath, attempt to evaluate them for expression language
+final PropertyDescriptor descriptor = 
component.getPropertyDescriptor(entry.getKey());
+if (descriptor.isDynamicClasspathModifier() && 
!StringUtils.isEmpty(entry.getValue())) {
+final StandardPropertyValue propertyValue = 
new StandardPropertyValue(entry.getValue(), null, variableRegistry);
+
modulePaths.add(propertyValue.evaluateAttributeExpressions().getValue());
 }
 }
+}
 
-try {
-component.onPropertyModified(descriptor, oldValue, 
value);
-} catch (final Exception e) {
-// nothing really to do here...
-}
+final boolean requiresInstanceClassLoading = 
ExtensionManager.requiresInstanceClassLoading(component.getClass().getTypeName());
+if (requiresInstanceClassLoading) {
+processClasspathModifiers(modulePaths);
+} else if (!requiresInstanceClassLoading && 
modulePaths.size() > 0) {
+// Component has property descriptors with 
dynamicallyModifiesClasspath set to true, but the class does not have 
@RequiresInstanceClassLoading
+logger.warn("One or more property descriptors intend 
to modify the classpath, " +
+"but component does not contain the {} 
annotation, classpath will not be modified",
+new Object[] 
{RequiresInstanceClassLoading.class.getName()});
 }
 }
 } finally {
 lock.unlock();
 }
 }
 
+// Keep setProperty/removeProperty private so that all calls go 
through setProperties
+private void setProperty(final String name, final String value) {
+if (null == name || null == value) {
+throw new IllegalArgumentException();
+}
--- End diff --

I see, but I still think we have to fix it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but 

[jira] [Commented] (NIFI-1322) Support appending files in PutHDFS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636057#comment-15636057
 ] 

ASF GitHub Bot commented on NIFI-1322:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1181#discussion_r86526545
  
--- Diff: 
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
 ---
@@ -74,9 +75,15 @@
 @SeeAlso(GetHDFS.class)
 public class PutHDFS extends AbstractHadoopProcessor {
 
-public static final String REPLACE_RESOLUTION = "replace";
-public static final String IGNORE_RESOLUTION = "ignore";
-public static final String FAIL_RESOLUTION = "fail";
--- End diff --

Oh yes, you're right! Will update this part asap!


> Support appending files in PutHDFS
> --
>
> Key: NIFI-1322
> URL: https://issues.apache.org/jira/browse/NIFI-1322
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Scott
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1181: NIFI-1322 - PutHDFS - allow file append resolution

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1181#discussion_r86526545
  
--- Diff: 
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
 ---
@@ -74,9 +75,15 @@
 @SeeAlso(GetHDFS.class)
 public class PutHDFS extends AbstractHadoopProcessor {
 
-public static final String REPLACE_RESOLUTION = "replace";
-public static final String IGNORE_RESOLUTION = "ignore";
-public static final String FAIL_RESOLUTION = "fail";
--- End diff --

Oh yes, you're right! Will update this part asap!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2909) Provide a framework mechanism for loading additional classpath resources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636042#comment-15636042
 ] 

ASF GitHub Bot commented on NIFI-2909:
--

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1156#discussion_r86525918
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/file/classloader/ClassLoaderUtils.java
 ---
@@ -52,23 +85,33 @@ public static ClassLoader getCustomClassLoader(String 
modulePath, ClassLoader pa
 isUrl = false;
 }
 if (!isUrl) {
-File modulePath = new File(modulePathString);
+try {
+File modulePath = new File(modulePathString);
 
-if (modulePath.exists()) {
+if (modulePath.exists()) {
 
-
additionalClasspath.add(modulePath.toURI().toURL());
+
additionalClasspath.add(modulePath.toURI().toURL());
 
-if (modulePath.isDirectory()) {
-File[] files = 
modulePath.listFiles(filenameFilter);
+if (modulePath.isDirectory()) {
+File[] files = 
modulePath.listFiles(filenameFilter);
 
-if (files != null) {
-for (File jarFile : files) {
-
additionalClasspath.add(jarFile.toURI().toURL());
+if (files != null) {
+for (File jarFile : files) {
--- End diff --

Since this one is really a stylistic comment, will leave it up to you, but. 
. .
After the change to determine if it's a directory, calling it _jarFile_ 
doesn't make much sense. Also, given that claspath can also have non-JAR 
resources (e.g., configuration, property files etc.) I think simply calling it 
something like _cpEntry_ or _classpathEntry_ would be more appropriate. 




> Provide a framework mechanism for loading additional classpath resources
> 
>
> Key: NIFI-2909
> URL: https://issues.apache.org/jira/browse/NIFI-2909
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
> Fix For: 1.1.0
>
>
> We currently have several components with a property for specifying 
> additional classpath resources (DBCP connection pool, scripting processors, 
> JMS). Each of these components is responsible for handling this in its own 
> way. 
> The framework should provide a more integrated solution to make it easier for 
> component developers to deal with this scenario. Some requirements that need 
> to be met by this solution:
> - Multiple instances of the same component with different resources added to 
> the classpath and not interfering with each other (i.e. two DBCP connection 
> pools using different drivers)
> - Ability to modify the actual ClassLoader of the component to deal with 
> frameworks that use Class.forName() without passing in a ClassLoader, meaning 
> if a processor loads class A and class A calls Class.forName(classBName), 
> then class B needs to be available in the ClassLoader that loaded the 
> processor's class which in turn loaded class A
> - A component developer should be able to indicate that a given 
> PropertyDescriptor represents a classpath resource and the framework should 
> take care of the ClassLoader manipulation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1156: NIFI-2909 and NIFI-1712 Per-Instance ClassLoading

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1156#discussion_r86525918
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/file/classloader/ClassLoaderUtils.java
 ---
@@ -52,23 +85,33 @@ public static ClassLoader getCustomClassLoader(String 
modulePath, ClassLoader pa
 isUrl = false;
 }
 if (!isUrl) {
-File modulePath = new File(modulePathString);
+try {
+File modulePath = new File(modulePathString);
 
-if (modulePath.exists()) {
+if (modulePath.exists()) {
 
-
additionalClasspath.add(modulePath.toURI().toURL());
+
additionalClasspath.add(modulePath.toURI().toURL());
 
-if (modulePath.isDirectory()) {
-File[] files = 
modulePath.listFiles(filenameFilter);
+if (modulePath.isDirectory()) {
+File[] files = 
modulePath.listFiles(filenameFilter);
 
-if (files != null) {
-for (File jarFile : files) {
-
additionalClasspath.add(jarFile.toURI().toURL());
+if (files != null) {
+for (File jarFile : files) {
--- End diff --

Since this one is really a stylistic comment, will leave it up to you, but. 
. .
After the change to determine if it's a directory, calling it _jarFile_ 
doesn't make much sense. Also, given that claspath can also have non-JAR 
resources (e.g., configuration, property files etc.) I think simply calling it 
something like _cpEntry_ or _classpathEntry_ would be more appropriate. 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2948) AWS DynamoDB Processors Do Not Update Credentials

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636026#comment-15636026
 ] 

ASF GitHub Bot commented on NIFI-2948:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1182


> AWS DynamoDB Processors Do Not Update Credentials
> -
>
> Key: NIFI-2948
> URL: https://issues.apache.org/jira/browse/NIFI-2948
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.1.0
>
>
> Following a report to the dev email list, there appears to be a bug in the 
> AWS DynamoDB processors where the AWS credentials are cached by the processor 
> once started, and not updated even if the processor is stopped, reconfigured, 
> and started.
> The problem appears to be 
> [AbstractDynamoDBProcessor::getDynamoDB|https://github.com/apache/nifi/blob/rel/nifi-1.0.0/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/dynamodb/AbstractDynamoDBProcessor.java#L218],
>  which assigns the dynamo client object to a class variable only once.
> *Repro Steps*
> # Add GetDynamoDB processor
> # Configure GetDynamoDB required properties but *not* any credential fields
> # Start the processor
> # Run a flowfile through - it should fail with 
> "MissingAuthenticationTokenException" error
> # Stop the GetDynamoDB processor
> # Configure valid AWS credentials
> # Start the processor
> # Run a flowfile through - it should fail again with 
> "MissingAuthenticationTokenException" still using the old credentials



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1182: NIFI-2948 Fix DynamoDB Cached Credentials

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1182


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2948) AWS DynamoDB Processors Do Not Update Credentials

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636023#comment-15636023
 ] 

ASF subversion and git services commented on NIFI-2948:
---

Commit 50010fb34064d9cf622e265ed57c9ab0be39318e in nifi's branch 
refs/heads/master from [~jameswing]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=50010fb ]

NIFI-2948 Fix DynamoDB Cached Credentials

This closes #1182


> AWS DynamoDB Processors Do Not Update Credentials
> -
>
> Key: NIFI-2948
> URL: https://issues.apache.org/jira/browse/NIFI-2948
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: James Wing
>Priority: Minor
> Fix For: 1.1.0
>
>
> Following a report to the dev email list, there appears to be a bug in the 
> AWS DynamoDB processors where the AWS credentials are cached by the processor 
> once started, and not updated even if the processor is stopped, reconfigured, 
> and started.
> The problem appears to be 
> [AbstractDynamoDBProcessor::getDynamoDB|https://github.com/apache/nifi/blob/rel/nifi-1.0.0/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/dynamodb/AbstractDynamoDBProcessor.java#L218],
>  which assigns the dynamo client object to a class variable only once.
> *Repro Steps*
> # Add GetDynamoDB processor
> # Configure GetDynamoDB required properties but *not* any credential fields
> # Start the processor
> # Run a flowfile through - it should fail with 
> "MissingAuthenticationTokenException" error
> # Stop the GetDynamoDB processor
> # Configure valid AWS credentials
> # Start the processor
> # Run a flowfile through - it should fail again with 
> "MissingAuthenticationTokenException" still using the old credentials



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2948) AWS DynamoDB Processors Do Not Update Credentials

[Oleg Zhurakousky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky updated NIFI-2948:
---
Assignee: James Wing

> AWS DynamoDB Processors Do Not Update Credentials
> -
>
> Key: NIFI-2948
> URL: https://issues.apache.org/jira/browse/NIFI-2948
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.1.0
>
>
> Following a report to the dev email list, there appears to be a bug in the 
> AWS DynamoDB processors where the AWS credentials are cached by the processor 
> once started, and not updated even if the processor is stopped, reconfigured, 
> and started.
> The problem appears to be 
> [AbstractDynamoDBProcessor::getDynamoDB|https://github.com/apache/nifi/blob/rel/nifi-1.0.0/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/dynamodb/AbstractDynamoDBProcessor.java#L218],
>  which assigns the dynamo client object to a class variable only once.
> *Repro Steps*
> # Add GetDynamoDB processor
> # Configure GetDynamoDB required properties but *not* any credential fields
> # Start the processor
> # Run a flowfile through - it should fail with 
> "MissingAuthenticationTokenException" error
> # Stop the GetDynamoDB processor
> # Configure valid AWS credentials
> # Start the processor
> # Run a flowfile through - it should fail again with 
> "MissingAuthenticationTokenException" still using the old credentials



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2948) AWS DynamoDB Processors Do Not Update Credentials

[Oleg Zhurakousky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky updated NIFI-2948:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> AWS DynamoDB Processors Do Not Update Credentials
> -
>
> Key: NIFI-2948
> URL: https://issues.apache.org/jira/browse/NIFI-2948
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.1.0
>
>
> Following a report to the dev email list, there appears to be a bug in the 
> AWS DynamoDB processors where the AWS credentials are cached by the processor 
> once started, and not updated even if the processor is stopped, reconfigured, 
> and started.
> The problem appears to be 
> [AbstractDynamoDBProcessor::getDynamoDB|https://github.com/apache/nifi/blob/rel/nifi-1.0.0/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/dynamodb/AbstractDynamoDBProcessor.java#L218],
>  which assigns the dynamo client object to a class variable only once.
> *Repro Steps*
> # Add GetDynamoDB processor
> # Configure GetDynamoDB required properties but *not* any credential fields
> # Start the processor
> # Run a flowfile through - it should fail with 
> "MissingAuthenticationTokenException" error
> # Stop the GetDynamoDB processor
> # Configure valid AWS credentials
> # Start the processor
> # Run a flowfile through - it should fail again with 
> "MissingAuthenticationTokenException" still using the old credentials



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2948) AWS DynamoDB Processors Do Not Update Credentials

[Oleg Zhurakousky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky updated NIFI-2948:
---
Fix Version/s: 1.1.0

> AWS DynamoDB Processors Do Not Update Credentials
> -
>
> Key: NIFI-2948
> URL: https://issues.apache.org/jira/browse/NIFI-2948
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: James Wing
>Priority: Minor
> Fix For: 1.1.0
>
>
> Following a report to the dev email list, there appears to be a bug in the 
> AWS DynamoDB processors where the AWS credentials are cached by the processor 
> once started, and not updated even if the processor is stopped, reconfigured, 
> and started.
> The problem appears to be 
> [AbstractDynamoDBProcessor::getDynamoDB|https://github.com/apache/nifi/blob/rel/nifi-1.0.0/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/dynamodb/AbstractDynamoDBProcessor.java#L218],
>  which assigns the dynamo client object to a class variable only once.
> *Repro Steps*
> # Add GetDynamoDB processor
> # Configure GetDynamoDB required properties but *not* any credential fields
> # Start the processor
> # Run a flowfile through - it should fail with 
> "MissingAuthenticationTokenException" error
> # Stop the GetDynamoDB processor
> # Configure valid AWS credentials
> # Start the processor
> # Run a flowfile through - it should fail again with 
> "MissingAuthenticationTokenException" still using the old credentials



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1322) Support appending files in PutHDFS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636012#comment-15636012
 ] 

ASF GitHub Bot commented on NIFI-1322:
--

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1181#discussion_r86524045
  
--- Diff: 
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
 ---
@@ -315,21 +328,24 @@ public void process(InputStream in) throws 
IOException {
 final long millis = 
stopWatch.getDuration(TimeUnit.MILLISECONDS);
 tempDotCopyFile = tempCopyFile;
 
-boolean renamed = false;
-for (int i = 0; i < 10; i++) { // try to rename multiple times.
-if (hdfs.rename(tempCopyFile, copyFile)) {
-renamed = true;
-break;// rename was successful
+if(!conflictResponse.equals(APPEND_RESOLUTION.getValue())
+|| 
(conflictResponse.equals(APPEND_RESOLUTION.getValue()) && !destinationExists)) {
+boolean renamed = false;
+for (int i = 0; i < 10; i++) { // try to rename multiple 
times.
+if (hdfs.rename(tempCopyFile, copyFile)) {
+renamed = true;
+break;// rename was successful
+}
+Thread.sleep(200L);// try waiting to let whatever 
might cause rename failure to resolve
--- End diff --

Do you know what's this for and if we still need it? I mean not only the 
sleep, but he whole loop.


> Support appending files in PutHDFS
> --
>
> Key: NIFI-1322
> URL: https://issues.apache.org/jira/browse/NIFI-1322
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Scott
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2948) AWS DynamoDB Processors Do Not Update Credentials

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636016#comment-15636016
 ] 

ASF GitHub Bot commented on NIFI-2948:
--

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/1182
  
+1; Merging


> AWS DynamoDB Processors Do Not Update Credentials
> -
>
> Key: NIFI-2948
> URL: https://issues.apache.org/jira/browse/NIFI-2948
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: James Wing
>Priority: Minor
>
> Following a report to the dev email list, there appears to be a bug in the 
> AWS DynamoDB processors where the AWS credentials are cached by the processor 
> once started, and not updated even if the processor is stopped, reconfigured, 
> and started.
> The problem appears to be 
> [AbstractDynamoDBProcessor::getDynamoDB|https://github.com/apache/nifi/blob/rel/nifi-1.0.0/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/dynamodb/AbstractDynamoDBProcessor.java#L218],
>  which assigns the dynamo client object to a class variable only once.
> *Repro Steps*
> # Add GetDynamoDB processor
> # Configure GetDynamoDB required properties but *not* any credential fields
> # Start the processor
> # Run a flowfile through - it should fail with 
> "MissingAuthenticationTokenException" error
> # Stop the GetDynamoDB processor
> # Configure valid AWS credentials
> # Start the processor
> # Run a flowfile through - it should fail again with 
> "MissingAuthenticationTokenException" still using the old credentials



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1182: NIFI-2948 Fix DynamoDB Cached Credentials

[olegz]

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/1182
  
+1; Merging


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1322) Support appending files in PutHDFS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636011#comment-15636011
 ] 

ASF GitHub Bot commented on NIFI-1322:
--

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1181#discussion_r86523527
  
--- Diff: 
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
 ---
@@ -74,9 +75,15 @@
 @SeeAlso(GetHDFS.class)
 public class PutHDFS extends AbstractHadoopProcessor {
 
-public static final String REPLACE_RESOLUTION = "replace";
-public static final String IGNORE_RESOLUTION = "ignore";
-public static final String FAIL_RESOLUTION = "fail";
--- End diff --

Pierre; I understand why/what you did here, but given the fact that both 
the class and the variable names are ```public```, this constitutes a breaking 
change.


> Support appending files in PutHDFS
> --
>
> Key: NIFI-1322
> URL: https://issues.apache.org/jira/browse/NIFI-1322
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Scott
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1181: NIFI-1322 - PutHDFS - allow file append resolution

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1181#discussion_r86524045
  
--- Diff: 
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
 ---
@@ -315,21 +328,24 @@ public void process(InputStream in) throws 
IOException {
 final long millis = 
stopWatch.getDuration(TimeUnit.MILLISECONDS);
 tempDotCopyFile = tempCopyFile;
 
-boolean renamed = false;
-for (int i = 0; i < 10; i++) { // try to rename multiple times.
-if (hdfs.rename(tempCopyFile, copyFile)) {
-renamed = true;
-break;// rename was successful
+if(!conflictResponse.equals(APPEND_RESOLUTION.getValue())
+|| 
(conflictResponse.equals(APPEND_RESOLUTION.getValue()) && !destinationExists)) {
+boolean renamed = false;
+for (int i = 0; i < 10; i++) { // try to rename multiple 
times.
+if (hdfs.rename(tempCopyFile, copyFile)) {
+renamed = true;
+break;// rename was successful
+}
+Thread.sleep(200L);// try waiting to let whatever 
might cause rename failure to resolve
--- End diff --

Do you know what's this for and if we still need it? I mean not only the 
sleep, but he whole loop.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1181: NIFI-1322 - PutHDFS - allow file append resolution

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1181#discussion_r86523527
  
--- Diff: 
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
 ---
@@ -74,9 +75,15 @@
 @SeeAlso(GetHDFS.class)
 public class PutHDFS extends AbstractHadoopProcessor {
 
-public static final String REPLACE_RESOLUTION = "replace";
-public static final String IGNORE_RESOLUTION = "ignore";
-public static final String FAIL_RESOLUTION = "fail";
--- End diff --

Pierre; I understand why/what you did here, but given the fact that both 
the class and the variable names are ```public```, this constitutes a breaking 
change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---