[jira] [Commented] (NIFI-3414) Implement an EnforceOrder processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860874#comment-15860874
 ] 

ASF GitHub Bot commented on NIFI-3414:
--

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/1496

NIFI-3414: Added EnforceOrder processor

Use it with FirstInFirstOutPrioritizer, it can enforce original ordering
of 'out-of-order' FlowFiles.

nifi-mock is modified to support FlowFile assertion using Prioritizer.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-3414

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1496.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1496


commit a60fc5d88e1a1ab3190052f19e228908e996c818
Author: Koji Kawamura 
Date:   2017-02-07T15:13:23Z

NIFI-3414: Added EnforceOrder processor

Use it with FirstInFirstOutPrioritizer, it can enforce original ordering
of 'out-of-order' FlowFiles.

nifi-mock is modified to support FlowFile assertion using Prioritizer.




> Implement an EnforceOrder processor
> ---
>
> Key: NIFI-3414
> URL: https://issues.apache.org/jira/browse/NIFI-3414
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Matt Burgess
>Assignee: Koji Kawamura
>
> For some flows, it is imperative that the flow files are processed in a 
> certain order.  The PriorityAttributePrioritizer can be used on a connection 
> to ensure that flow files going through that connection are in priority 
> order, but depending on error-handling, branching, and other flow designs, it 
> is possible for flow files to get out-of-order.
> I propose an EnforceOrder processor, which would be single-threaded and have 
> (at a minimum) the following properties:
> 1) Order Attribute: This would be the name of a flow file attribute from 
> which the current value will be retrieved.
> 2) Initial Value: This property specifies an initial value for the order. The 
> processor is stateful, however, so this property is only used when there is 
> no entry in the state map for current value.
> The processor would store the Initial Value into the state map (if no state 
> map entry exists), then for each incoming flow file, it checks the value in 
> the Order Attribute against the current value.  If the attribute value 
> matches the current value, the flow file is transferred to the "success" 
> relationship, and the current value is incremented in the state map. If the 
> attribute value does not match the current value, the session will be rolled 
> back.
> Using this processor, along with a PriorityAttributePrioritizer on the 
> incoming connection, will allow for out-of-order flow files to have a sort of 
> "barrier", thereby guaranteeing that flow files transferred to the "success" 
> 

[GitHub] nifi pull request #1496: NIFI-3414: Added EnforceOrder processor

[ijokarumawak]

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/1496

NIFI-3414: Added EnforceOrder processor

Use it with FirstInFirstOutPrioritizer, it can enforce original ordering
of 'out-of-order' FlowFiles.

nifi-mock is modified to support FlowFile assertion using Prioritizer.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-3414

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1496.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1496


commit a60fc5d88e1a1ab3190052f19e228908e996c818
Author: Koji Kawamura 
Date:   2017-02-07T15:13:23Z

NIFI-3414: Added EnforceOrder processor

Use it with FirstInFirstOutPrioritizer, it can enforce original ordering
of 'out-of-order' FlowFiles.

nifi-mock is modified to support FlowFile assertion using Prioritizer.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3449) Create Google Cloud Platform/Google Cloud Storage Processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860782#comment-15860782
 ] 

ASF GitHub Bot commented on NIFI-3449:
--

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100469820
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/ListGCSBucket.java
 ---
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.storage;
+
+import com.google.cloud.Page;
+import com.google.cloud.storage.Acl;
+import com.google.cloud.storage.Blob;
+import com.google.cloud.storage.BlobInfo;
+import com.google.cloud.storage.Storage;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CREATE_TIME_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CREATE_TIME_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_ALGORITHM_ATTR;
+import static 

[jira] [Commented] (NIFI-3449) Create Google Cloud Platform/Google Cloud Storage Processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860789#comment-15860789
 ] 

ASF GitHub Bot commented on NIFI-3449:
--

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100469527
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/FetchGCSObject.java
 ---
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.storage;
+
+import com.google.cloud.ReadChannel;
+import com.google.cloud.storage.Acl;
+import com.google.cloud.storage.Blob;
+import com.google.cloud.storage.BlobId;
+import com.google.cloud.storage.BlobInfo;
+import com.google.cloud.storage.Storage;
+import com.google.cloud.storage.StorageException;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.nio.channels.Channels;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CREATE_TIME_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CREATE_TIME_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_ALGORITHM_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_ALGORITHM_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_SHA256_ATTR;
+import static 

[jira] [Commented] (NIFI-3449) Create Google Cloud Platform/Google Cloud Storage Processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860787#comment-15860787
 ] 

ASF GitHub Bot commented on NIFI-3449:
--

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100471880
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/PutGCSObject.java
 ---
@@ -0,0 +1,523 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.storage;
+
+import com.google.cloud.storage.Acl;
+import com.google.cloud.storage.Blob;
+import com.google.cloud.storage.BlobId;
+import com.google.cloud.storage.BlobInfo;
+import com.google.cloud.storage.Storage;
+import com.google.cloud.storage.StorageException;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.ALL_AUTHENTICATED_USERS;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.AUTHENTICATED_READ;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.BUCKET_OWNER_FULL_CONTROL;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.BUCKET_OWNER_READ;
+import static com.google.cloud.storage.Storage.PredefinedAcl.PRIVATE;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.PROJECT_PRIVATE;
+import static com.google.cloud.storage.Storage.PredefinedAcl.PUBLIC_READ;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_DESC;
+import static 

[jira] [Commented] (NIFI-3449) Create Google Cloud Platform/Google Cloud Storage Processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860790#comment-15860790
 ] 

ASF GitHub Bot commented on NIFI-3449:
--

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100464227
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/factory/strategies/JsonFileServiceAccountCredentialsStrategy.java
 ---
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.credentials.factory.strategies;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import 
org.apache.nifi.processors.gcp.credentials.factory.CredentialPropertyDescriptors;
+import org.apache.nifi.stream.io.BufferedInputStream;
--- End diff --

Did you mean to use this deprecated 
`org.apache.nifi.stream.io.BufferedInputStream`?  It looks like something your 
IDE might have "helpfully" chosen instead of `java.io.BufferedInputStream`.


> Create Google Cloud Platform/Google Cloud Storage Processors
> 
>
> Key: NIFI-3449
> URL: https://issues.apache.org/jira/browse/NIFI-3449
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Gene Peters
>  Labels: features
>
> Hi all,
> We had a need in our production deployments to interact with Google Cloud 
> Storage. At the time, NIFI-2809 hadn't seen much movement, and after applying 
> the patch I found that the configuration was too specific for my needs (it's 
> hardcoded to use "Application Default" credentials, everything uploaded to 
> GCS is uploaded with the "public" ACL, etc). So I created a series of 
> Processors / Controller Services based off of the AWS NiFi library, and would 
> like to contribute them. 
> Features:
> * All credentialing is handled by a controller service, allowing multiple 
> processors to use the same service / credentials
> * An Abstract processor is provided which forms the basis for all GCP related 
> processors.
> * The standard Google Cloud Storage operations are supported, very similarly 
> to the AWS S3 processors: ListGCSBucket, DeleteGCSObject, FetchGCSObject, 
> PutGCSObject
> * Everything is documented and unit tested. 
> * I've also provided integration tests, but they're disabled by default (as 
> they require Google Cloud credentials). To run them, use the flag 
> {{skipGCPIntegrationTests=false}}
> Todo:
> * The GCP Java library's ReadChannel objects implement the "restorable" 
> interface, which allows for state saving / checkpointing. I'd really like to 
> leverage this with the State support that NiFi provides, but it would require 
> serializing / deserializing the object. 
> I'm going to be submitting this as a pull request through GitHub. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3449) Create Google Cloud Platform/Google Cloud Storage Processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860783#comment-15860783
 ] 

ASF GitHub Bot commented on NIFI-3449:
--

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100463954
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/factory/strategies/AbstractServiceAccountCredentialsStrategy.java
 ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.credentials.factory.strategies;
+
+import com.google.auth.oauth2.GoogleCredentials;
+import org.apache.nifi.components.PropertyDescriptor;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Map;
+
+/**
+ * Supports Google Cloud Application Default Credentials.
--- End diff --

This comment looks like a copy/paste error.


> Create Google Cloud Platform/Google Cloud Storage Processors
> 
>
> Key: NIFI-3449
> URL: https://issues.apache.org/jira/browse/NIFI-3449
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Gene Peters
>  Labels: features
>
> Hi all,
> We had a need in our production deployments to interact with Google Cloud 
> Storage. At the time, NIFI-2809 hadn't seen much movement, and after applying 
> the patch I found that the configuration was too specific for my needs (it's 
> hardcoded to use "Application Default" credentials, everything uploaded to 
> GCS is uploaded with the "public" ACL, etc). So I created a series of 
> Processors / Controller Services based off of the AWS NiFi library, and would 
> like to contribute them. 
> Features:
> * All credentialing is handled by a controller service, allowing multiple 
> processors to use the same service / credentials
> * An Abstract processor is provided which forms the basis for all GCP related 
> processors.
> * The standard Google Cloud Storage operations are supported, very similarly 
> to the AWS S3 processors: ListGCSBucket, DeleteGCSObject, FetchGCSObject, 
> PutGCSObject
> * Everything is documented and unit tested. 
> * I've also provided integration tests, but they're disabled by default (as 
> they require Google Cloud credentials). To run them, use the flag 
> {{skipGCPIntegrationTests=false}}
> Todo:
> * The GCP Java library's ReadChannel objects implement the "restorable" 
> interface, which allows for state saving / checkpointing. I'd really like to 
> leverage this with the State support that NiFi provides, but it would require 
> serializing / deserializing the object. 
> I'm going to be submitting this as a pull request through GitHub. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3449) Create Google Cloud Platform/Google Cloud Storage Processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860788#comment-15860788
 ] 

ASF GitHub Bot commented on NIFI-3449:
--

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100471948
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/PutGCSObject.java
 ---
@@ -0,0 +1,523 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.storage;
+
+import com.google.cloud.storage.Acl;
+import com.google.cloud.storage.Blob;
+import com.google.cloud.storage.BlobId;
+import com.google.cloud.storage.BlobInfo;
+import com.google.cloud.storage.Storage;
+import com.google.cloud.storage.StorageException;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.ALL_AUTHENTICATED_USERS;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.AUTHENTICATED_READ;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.BUCKET_OWNER_FULL_CONTROL;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.BUCKET_OWNER_READ;
+import static com.google.cloud.storage.Storage.PredefinedAcl.PRIVATE;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.PROJECT_PRIVATE;
+import static com.google.cloud.storage.Storage.PredefinedAcl.PUBLIC_READ;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_DESC;
+import static 

[jira] [Commented] (NIFI-3449) Create Google Cloud Platform/Google Cloud Storage Processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860785#comment-15860785
 ] 

ASF GitHub Bot commented on NIFI-3449:
--

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100469488
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/FetchGCSObject.java
 ---
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.storage;
+
+import com.google.cloud.ReadChannel;
+import com.google.cloud.storage.Acl;
+import com.google.cloud.storage.Blob;
+import com.google.cloud.storage.BlobId;
+import com.google.cloud.storage.BlobInfo;
+import com.google.cloud.storage.Storage;
+import com.google.cloud.storage.StorageException;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.nio.channels.Channels;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CREATE_TIME_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CREATE_TIME_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_ALGORITHM_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_ALGORITHM_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_SHA256_ATTR;
+import static 

[jira] [Commented] (NIFI-3449) Create Google Cloud Platform/Google Cloud Storage Processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860784#comment-15860784
 ] 

ASF GitHub Bot commented on NIFI-3449:
--

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100471403
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/factory/strategies/AbstractBooleanCredentialsStrategy.java
 ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.credentials.factory.strategies;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import 
org.apache.nifi.processors.gcp.credentials.factory.CredentialsStrategy;
+
+import java.util.ArrayList;
+import java.util.Collection;
+
+
+/**
+ * Partial implementation of CredentialsStrategy to provide support for 
credential strategies specified by
+ * a single boolean property.
+ */
+public abstract class AbstractBooleanCredentialsStrategy extends 
AbstractCredentialsStrategy {
--- End diff --

I think we need an override for `canCreatePrimaryCredential()` that parses 
the boolean property value.  As-is, I believe it fails in the following 
scenario:
1. Create a GCPCredentialsControllerService
2. Edit properties to set "Use Application Default Credentials" to `true`.  
Click Apply.
3. Edit properties to set "Use Application Default Credentials" to `false`, 
and "Use Compute Engine Credentials" to `true`.  Click Apply.
The controller service is now in an invalid state because "Use Application 
Default Credentials" thinks it's in a viable state with a non-null property 
value, without parsing the property to understand "false" means no.


> Create Google Cloud Platform/Google Cloud Storage Processors
> 
>
> Key: NIFI-3449
> URL: https://issues.apache.org/jira/browse/NIFI-3449
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Gene Peters
>  Labels: features
>
> Hi all,
> We had a need in our production deployments to interact with Google Cloud 
> Storage. At the time, NIFI-2809 hadn't seen much movement, and after applying 
> the patch I found that the configuration was too specific for my needs (it's 
> hardcoded to use "Application Default" credentials, everything uploaded to 
> GCS is uploaded with the "public" ACL, etc). So I created a series of 
> Processors / Controller Services based off of the AWS NiFi library, and would 
> like to contribute them. 
> Features:
> * All credentialing is handled by a controller service, allowing multiple 
> processors to use the same service / credentials
> * An Abstract processor is provided which forms the basis for all GCP related 
> processors.
> * The standard Google Cloud Storage operations are supported, very similarly 
> to the AWS S3 processors: ListGCSBucket, DeleteGCSObject, FetchGCSObject, 
> PutGCSObject
> * Everything is documented and unit tested. 
> * I've also provided integration tests, but they're disabled by default (as 
> they require Google Cloud credentials). To run them, use the flag 
> {{skipGCPIntegrationTests=false}}
> Todo:
> * The GCP Java library's ReadChannel objects implement the "restorable" 
> interface, which allows for state saving / checkpointing. I'd really like to 
> leverage this with the State support that NiFi provides, but it would require 
> serializing / deserializing the object. 
> I'm going to be submitting this as a pull request through GitHub. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1482: NIFI-3449: Adding GCP Framework and GCS Processors

[jvwing]

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100464227
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/factory/strategies/JsonFileServiceAccountCredentialsStrategy.java
 ---
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.credentials.factory.strategies;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import 
org.apache.nifi.processors.gcp.credentials.factory.CredentialPropertyDescriptors;
+import org.apache.nifi.stream.io.BufferedInputStream;
--- End diff --

Did you mean to use this deprecated 
`org.apache.nifi.stream.io.BufferedInputStream`?  It looks like something your 
IDE might have "helpfully" chosen instead of `java.io.BufferedInputStream`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1482: NIFI-3449: Adding GCP Framework and GCS Processors

[jvwing]

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100472194
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/StorageAttributes.java
 ---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.storage;
+
+
+/**
+ * Common attributes being written and accessed through Google Cloud 
Storage.
+ */
+public class StorageAttributes {
+private StorageAttributes() {}
+
+public static final String BUCKET_ATTR = "gcs.bucket";
+public static final String BUCKET_DESC = "Bucket of the object.";
+
+public static final String KEY_ATTR = "gcs.key";
+public static final String KEY_DESC = "Name of the object.";
+
+public static final String SIZE_ATTR = "gcs.size";
+public static final String SIZE_DESC = "Size of the object.";
+
+public static final String CACHE_CONTROL_ATTR = "gcs.cache.control";
+public static final String CACHE_CONTROL_DESC = "Data cache control of 
the object.";
+
+public static final String COMPONENT_COUNT_ATTR = 
"gcs.component.count";
+public static final String COMPONENT_COUNT_DESC = "The number of 
components which make up the object.";
+
+public static final String CONTENT_DISPOSITION_ATTR = 
"gcs.content.disposition";
+public static final String CONTENT_DISPOSITION_DESC = "The data 
content disposition of the object.";
+
+public static final String CONTENT_ENCODING_ATTR = 
"gcs.content.encoding";
+public static final String CONTENT_ENCODING_DESC = "The content 
encoding of the object.";
+
+public static final String CONTENT_LANGUAGE_ATTR = 
"gcs.content.language";
+public static final String CONTENT_LANGUAGE_DESC = "The content 
language of the object.";
+
+public static final String CONTENT_TYPE_ATTR = "gcs.content.type";
--- End diff --

Did you consider using `mime.type` here?  It is a mostly-standardized NiFi 
attribute for content types, widely used by other processors.  See 
[CoreAttributes](https://github.com/apache/nifi/blob/master/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/flowfile/attributes/CoreAttributes.java).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1482: NIFI-3449: Adding GCP Framework and GCS Processors

[jvwing]

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100471948
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/PutGCSObject.java
 ---
@@ -0,0 +1,523 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.storage;
+
+import com.google.cloud.storage.Acl;
+import com.google.cloud.storage.Blob;
+import com.google.cloud.storage.BlobId;
+import com.google.cloud.storage.BlobInfo;
+import com.google.cloud.storage.Storage;
+import com.google.cloud.storage.StorageException;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.ALL_AUTHENTICATED_USERS;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.AUTHENTICATED_READ;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.BUCKET_OWNER_FULL_CONTROL;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.BUCKET_OWNER_READ;
+import static com.google.cloud.storage.Storage.PredefinedAcl.PRIVATE;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.PROJECT_PRIVATE;
+import static com.google.cloud.storage.Storage.PredefinedAcl.PUBLIC_READ;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_ATTR;
+import static 

[GitHub] nifi pull request #1482: NIFI-3449: Adding GCP Framework and GCS Processors

[jvwing]

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100463954
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/factory/strategies/AbstractServiceAccountCredentialsStrategy.java
 ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.credentials.factory.strategies;
+
+import com.google.auth.oauth2.GoogleCredentials;
+import org.apache.nifi.components.PropertyDescriptor;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Map;
+
+/**
+ * Supports Google Cloud Application Default Credentials.
--- End diff --

This comment looks like a copy/paste error.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1482: NIFI-3449: Adding GCP Framework and GCS Processors

[jvwing]

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100469527
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/FetchGCSObject.java
 ---
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.storage;
+
+import com.google.cloud.ReadChannel;
+import com.google.cloud.storage.Acl;
+import com.google.cloud.storage.Blob;
+import com.google.cloud.storage.BlobId;
+import com.google.cloud.storage.BlobInfo;
+import com.google.cloud.storage.Storage;
+import com.google.cloud.storage.StorageException;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.nio.channels.Channels;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CREATE_TIME_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CREATE_TIME_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_ALGORITHM_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_ALGORITHM_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_SHA256_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_SHA256_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ETAG_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ETAG_DESC;
+import static 

[GitHub] nifi pull request #1482: NIFI-3449: Adding GCP Framework and GCS Processors

[jvwing]

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100469488
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/FetchGCSObject.java
 ---
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.storage;
+
+import com.google.cloud.ReadChannel;
+import com.google.cloud.storage.Acl;
+import com.google.cloud.storage.Blob;
+import com.google.cloud.storage.BlobId;
+import com.google.cloud.storage.BlobInfo;
+import com.google.cloud.storage.Storage;
+import com.google.cloud.storage.StorageException;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.nio.channels.Channels;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CREATE_TIME_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CREATE_TIME_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_ALGORITHM_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_ALGORITHM_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_SHA256_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_SHA256_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ETAG_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ETAG_DESC;
+import static 

[GitHub] nifi pull request #1482: NIFI-3449: Adding GCP Framework and GCS Processors

[jvwing]

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100469820
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/ListGCSBucket.java
 ---
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.storage;
+
+import com.google.cloud.Page;
+import com.google.cloud.storage.Acl;
+import com.google.cloud.storage.Blob;
+import com.google.cloud.storage.BlobInfo;
+import com.google.cloud.storage.Storage;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CREATE_TIME_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CREATE_TIME_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_ALGORITHM_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_ALGORITHM_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.ENCRYPTION_SHA256_ATTR;
+import static 

[GitHub] nifi pull request #1482: NIFI-3449: Adding GCP Framework and GCS Processors

[jvwing]

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100471403
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/credentials/factory/strategies/AbstractBooleanCredentialsStrategy.java
 ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.credentials.factory.strategies;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import 
org.apache.nifi.processors.gcp.credentials.factory.CredentialsStrategy;
+
+import java.util.ArrayList;
+import java.util.Collection;
+
+
+/**
+ * Partial implementation of CredentialsStrategy to provide support for 
credential strategies specified by
+ * a single boolean property.
+ */
+public abstract class AbstractBooleanCredentialsStrategy extends 
AbstractCredentialsStrategy {
--- End diff --

I think we need an override for `canCreatePrimaryCredential()` that parses 
the boolean property value.  As-is, I believe it fails in the following 
scenario:
1. Create a GCPCredentialsControllerService
2. Edit properties to set "Use Application Default Credentials" to `true`.  
Click Apply.
3. Edit properties to set "Use Application Default Credentials" to `false`, 
and "Use Compute Engine Credentials" to `true`.  Click Apply.
The controller service is now in an invalid state because "Use Application 
Default Credentials" thinks it's in a viable state with a non-null property 
value, without parsing the property to understand "false" means no.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1482: NIFI-3449: Adding GCP Framework and GCS Processors

[jvwing]

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100471880
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/PutGCSObject.java
 ---
@@ -0,0 +1,523 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp.storage;
+
+import com.google.cloud.storage.Acl;
+import com.google.cloud.storage.Blob;
+import com.google.cloud.storage.BlobId;
+import com.google.cloud.storage.BlobInfo;
+import com.google.cloud.storage.Storage;
+import com.google.cloud.storage.StorageException;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.ALL_AUTHENTICATED_USERS;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.AUTHENTICATED_READ;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.BUCKET_OWNER_FULL_CONTROL;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.BUCKET_OWNER_READ;
+import static com.google.cloud.storage.Storage.PredefinedAcl.PRIVATE;
+import static 
com.google.cloud.storage.Storage.PredefinedAcl.PROJECT_PRIVATE;
+import static com.google.cloud.storage.Storage.PredefinedAcl.PUBLIC_READ;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.BUCKET_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CACHE_CONTROL_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.COMPONENT_COUNT_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_DISPOSITION_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_ENCODING_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_LANGUAGE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_ATTR;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CONTENT_TYPE_DESC;
+import static 
org.apache.nifi.processors.gcp.storage.StorageAttributes.CRC32C_ATTR;
+import static 

[jira] [Commented] (NIFI-3449) Create Google Cloud Platform/Google Cloud Storage Processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860779#comment-15860779
 ] 

ASF GitHub Bot commented on NIFI-3449:
--

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/1482
  
It looks like the unit tests are failing with the addition of expression 
language to properties in AbstractGCPProcessor.  I thought it was just me, but 
TravisCI seems to have the same issue:


>testDeleteWithValidArguments(org.apache.nifi.processors.gcp.storage.DeleteGCSObjectTest)
  Time elapsed: 1.479 sec  <<< FAILURE!
java.lang.AssertionError: Could not invoke methods annotated with 
@OnScheduled annotation due to: java.lang.reflect.InvocationTargetException
at 
org.apache.nifi.processors.gcp.storage.DeleteGCSObjectTest.testDeleteWithValidArguments(DeleteGCSObjectTest.java:82)

And many similar tests.  I know commenting out AbstractGCPProcessor lines 
45 and 55 makes the tests pass again.  I think the problem is we need to call 
`evaluateAttributeExpressions()`, but I'm not 100% on that.


> Create Google Cloud Platform/Google Cloud Storage Processors
> 
>
> Key: NIFI-3449
> URL: https://issues.apache.org/jira/browse/NIFI-3449
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Gene Peters
>  Labels: features
>
> Hi all,
> We had a need in our production deployments to interact with Google Cloud 
> Storage. At the time, NIFI-2809 hadn't seen much movement, and after applying 
> the patch I found that the configuration was too specific for my needs (it's 
> hardcoded to use "Application Default" credentials, everything uploaded to 
> GCS is uploaded with the "public" ACL, etc). So I created a series of 
> Processors / Controller Services based off of the AWS NiFi library, and would 
> like to contribute them. 
> Features:
> * All credentialing is handled by a controller service, allowing multiple 
> processors to use the same service / credentials
> * An Abstract processor is provided which forms the basis for all GCP related 
> processors.
> * The standard Google Cloud Storage operations are supported, very similarly 
> to the AWS S3 processors: ListGCSBucket, DeleteGCSObject, FetchGCSObject, 
> PutGCSObject
> * Everything is documented and unit tested. 
> * I've also provided integration tests, but they're disabled by default (as 
> they require Google Cloud credentials). To run them, use the flag 
> {{skipGCPIntegrationTests=false}}
> Todo:
> * The GCP Java library's ReadChannel objects implement the "restorable" 
> interface, which allows for state saving / checkpointing. I'd really like to 
> leverage this with the State support that NiFi provides, but it would require 
> serializing / deserializing the object. 
> I'm going to be submitting this as a pull request through GitHub. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1482: NIFI-3449: Adding GCP Framework and GCS Processors

[jvwing]

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/1482
  
It looks like the unit tests are failing with the addition of expression 
language to properties in AbstractGCPProcessor.  I thought it was just me, but 
TravisCI seems to have the same issue:


>testDeleteWithValidArguments(org.apache.nifi.processors.gcp.storage.DeleteGCSObjectTest)
  Time elapsed: 1.479 sec  <<< FAILURE!
java.lang.AssertionError: Could not invoke methods annotated with 
@OnScheduled annotation due to: java.lang.reflect.InvocationTargetException
at 
org.apache.nifi.processors.gcp.storage.DeleteGCSObjectTest.testDeleteWithValidArguments(DeleteGCSObjectTest.java:82)

And many similar tests.  I know commenting out AbstractGCPProcessor lines 
45 and 55 makes the tests pass again.  I think the problem is we need to call 
`evaluateAttributeExpressions()`, but I'm not 100% on that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3449) Create Google Cloud Platform/Google Cloud Storage Processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860643#comment-15860643
 ] 

ASF GitHub Bot commented on NIFI-3449:
--

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100466820
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java
 ---
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp;
+
+import com.google.auth.oauth2.GoogleCredentials;
+import com.google.cloud.HttpServiceOptions;
+import com.google.cloud.Service;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsService;
+
+import java.util.List;
+
+/**
+ * Abstract base class for gcp processors.
+ *
+ */
+public abstract class AbstractGCPProcessor<
+CloudService extends Service,
+CloudServiceRpc,
+CloudServiceOptions extends HttpServiceOptions> extends AbstractProcessor {
+
+public static final PropertyDescriptor PROJECT_ID = new 
PropertyDescriptor
+.Builder().name("gcp-project-id")
+.displayName("Project ID")
+.description("Google Cloud Project ID")
+.expressionLanguageSupported(true)
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor RETRY_COUNT = new 
PropertyDescriptor
+.Builder().name("gcp-retry-count")
+.displayName("Number of retries")
+.description("How many retry attempts should be made before 
routing to the failure relationship.")
+.defaultValue("6")
+.expressionLanguageSupported(true)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.build();
+
+/**
+ * Links to the {@link GCPCredentialsService} which provides 
credentials for this particular processor.
+ */
+public static final PropertyDescriptor 
GCP_CREDENTIALS_PROVIDER_SERVICE = new PropertyDescriptor.Builder()
+.name("gcp-credentials-provider-service")
+.name("GCP Credentials Provider Service")
+.description("The Controller Service used to obtain Google 
Cloud Platform credentials.")
+.required(true)
+.identifiesControllerService(GCPCredentialsService.class)
+.build();
+
+
+protected volatile CloudService cloudService;
+
+protected CloudService getCloudService() {
+return cloudService;
+}
+
+@Override
+public List getSupportedPropertyDescriptors() {
+return ImmutableList.of(
+GCP_CREDENTIALS_PROVIDER_SERVICE,
+PROJECT_ID,
+RETRY_COUNT
+);
+}
+
+
+/**
+ * Retrieve credentials from the {@link GCPCredentialsService} 
attached to this processor.
+ * @param context the process context provided on scheduling the 
processor.
+ * @return GoogleCredentials for the processor to access.
+ * @see  https://developers.google.com/api-client-library/java/google-api-java-client/reference/1.20.0/com/google/api/client/googleapis/auth/oauth2/GoogleCredential;>AuthCredentials
+ */
+private GoogleCredentials getGoogleCredentials(final ProcessContext 

[GitHub] nifi pull request #1482: NIFI-3449: Adding GCP Framework and GCS Processors

[joewitt]

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1482#discussion_r100466820
  
--- Diff: 
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java
 ---
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.gcp;
+
+import com.google.auth.oauth2.GoogleCredentials;
+import com.google.cloud.HttpServiceOptions;
+import com.google.cloud.Service;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsService;
+
+import java.util.List;
+
+/**
+ * Abstract base class for gcp processors.
+ *
+ */
+public abstract class AbstractGCPProcessor<
+CloudService extends Service,
+CloudServiceRpc,
+CloudServiceOptions extends HttpServiceOptions> extends AbstractProcessor {
+
+public static final PropertyDescriptor PROJECT_ID = new 
PropertyDescriptor
+.Builder().name("gcp-project-id")
+.displayName("Project ID")
+.description("Google Cloud Project ID")
+.expressionLanguageSupported(true)
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor RETRY_COUNT = new 
PropertyDescriptor
+.Builder().name("gcp-retry-count")
+.displayName("Number of retries")
+.description("How many retry attempts should be made before 
routing to the failure relationship.")
+.defaultValue("6")
+.expressionLanguageSupported(true)
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.build();
+
+/**
+ * Links to the {@link GCPCredentialsService} which provides 
credentials for this particular processor.
+ */
+public static final PropertyDescriptor 
GCP_CREDENTIALS_PROVIDER_SERVICE = new PropertyDescriptor.Builder()
+.name("gcp-credentials-provider-service")
+.name("GCP Credentials Provider Service")
+.description("The Controller Service used to obtain Google 
Cloud Platform credentials.")
+.required(true)
+.identifiesControllerService(GCPCredentialsService.class)
+.build();
+
+
+protected volatile CloudService cloudService;
+
+protected CloudService getCloudService() {
+return cloudService;
+}
+
+@Override
+public List getSupportedPropertyDescriptors() {
+return ImmutableList.of(
+GCP_CREDENTIALS_PROVIDER_SERVICE,
+PROJECT_ID,
+RETRY_COUNT
+);
+}
+
+
+/**
+ * Retrieve credentials from the {@link GCPCredentialsService} 
attached to this processor.
+ * @param context the process context provided on scheduling the 
processor.
+ * @return GoogleCredentials for the processor to access.
+ * @see  https://developers.google.com/api-client-library/java/google-api-java-client/reference/1.20.0/com/google/api/client/googleapis/auth/oauth2/GoogleCredential;>AuthCredentials
+ */
+private GoogleCredentials getGoogleCredentials(final ProcessContext 
context) {
+final GCPCredentialsService gcpCredentialsService =
+
context.getProperty(GCP_CREDENTIALS_PROVIDER_SERVICE).asControllerService(GCPCredentialsService.class);
+return 

[jira] [Commented] (NIFI-3356) Provide a newly refactored provenance repository

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860626#comment-15860626
 ] 

ASF GitHub Bot commented on NIFI-3356:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/1493
  
am a +1 to mike's point.  @markap14 can you please review/merge if 
appropriate 1475.  i'd be happy to dive through this once that is in place but 
agree we should not push this and complicate that one first.


> Provide a newly refactored provenance repository
> 
>
> Key: NIFI-3356
> URL: https://issues.apache.org/jira/browse/NIFI-3356
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> The Persistent Provenance Repository has been redesigned a few different 
> times over several years. The original design for the repository was to 
> provide storage of events and sequential iteration over those events via a 
> Reporting Task. After that, we added the ability to compress the data so that 
> it could be held longer. We then introduced the notion of indexing and 
> searching via Lucene. We've since made several more modifications to try to 
> boost performance.
> At this point, however, the repository is still the bottleneck for many flows 
> that handle large volumes of small FlowFiles. We need a new implementation 
> that is based around the current goals for the repository and that can 
> provide better throughput.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1493: NIFI-3356: Initial implementation of writeahead provenance...

[joewitt]

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/1493
  
am a +1 to mike's point.  @markap14 can you please review/merge if 
appropriate 1475.  i'd be happy to dive through this once that is in place but 
agree we should not push this and complicate that one first.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #42: MINIFI-186 Updating Copyright statements to refle...

[phrocker]

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/42
  
+1 totally a fan of 2017


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-3460) Statsd Reporting Task

[Nicholas Carenza (JIRA)]

Nicholas Carenza created NIFI-3460:
--

 Summary: Statsd Reporting Task
 Key: NIFI-3460
 URL: https://issues.apache.org/jira/browse/NIFI-3460
 Project: Apache NiFi
  Issue Type: New Feature
Reporter: Nicholas Carenza
Priority: Minor


I would like to be able to report metrics to Statsd.

https://github.com/etsy/statsd.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3460) Statsd Reporting Task

[Nicholas Carenza (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicholas Carenza updated NIFI-3460:
---
Labels: reporting_task  (was: )

> Statsd Reporting Task
> -
>
> Key: NIFI-3460
> URL: https://issues.apache.org/jira/browse/NIFI-3460
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Reporter: Nicholas Carenza
>Priority: Minor
>  Labels: reporting_task
>
> I would like to be able to report metrics to Statsd.
> https://github.com/etsy/statsd.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3460) Statsd Reporting Task

[Nicholas Carenza (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicholas Carenza updated NIFI-3460:
---
Component/s: Core Framework

> Statsd Reporting Task
> -
>
> Key: NIFI-3460
> URL: https://issues.apache.org/jira/browse/NIFI-3460
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Reporter: Nicholas Carenza
>Priority: Minor
>  Labels: reporting_task
>
> I would like to be able to report metrics to Statsd.
> https://github.com/etsy/statsd.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3454) Tests in TestClassloaderUtils should consistently use the FileNameFilter when using directories

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860315#comment-15860315
 ] 

Otto Fowler commented on NIFI-3454:
---

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 
2015-11-10T11:41:47-05:00)
Maven home: /usr/local/Cellar/maven/3.3.9/libexec
Java version: 1.8.0_31, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_31.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.12.3", arch: "x86_64", family: "mac"

> Tests in TestClassloaderUtils should consistently use the FileNameFilter when 
> using directories
> ---
>
> Key: NIFI-3454
> URL: https://issues.apache.org/jira/browse/NIFI-3454
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
> Environment: Mac OS X ( any version )  Java 1.8.0_31  Maven 3.3.9
>Reporter: Otto Fowler
>
> DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in 
> any directory on Mac OS X. 
> Any call made to getURLsForClasspath  or getCustomClassLoader passing 
> directories an not paths to jars for modules will return these files as urls 
> unless they are specifically filtered out using the FileNameFilter parameter.
> The FileNameFilter parameter to these functions can be used to protect 
> against this, and be sure the URL's returned are for the expected items.  In 
> fact there is a function  getJarFilenameFilter in the TestClassloaderUtils 
> already, that is used by the getCustomClassloader tests.
> This function should be used in all of the tests that pass in directories and 
> not paths that are to jar files.  Without it, NiFi builds that enable tests 
> will fail on Mac OS X if .DS_Store files exist in the testing directory.
> This problem also effects TestIdentifyMimeType.java, where it is doing a 
> directory listing, and should ignore "." files.
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3454) Tests in TestClassloaderUtils should consistently use the FileNameFilter when using directories

[Otto Fowler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler updated NIFI-3454:
--
Description: 
DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in any 
directory on Mac OS X. 

Any call made to getURLsForClasspath  or getCustomClassLoader passing 
directories an not paths to jars for modules will return these files as urls 
unless they are specifically filtered out using the FileNameFilter parameter.

The FileNameFilter parameter to these functions can be used to protect against 
this, and be sure the URL's returned are for the expected items.  In fact there 
is a function  getJarFilenameFilter in the TestClassloaderUtils already, that 
is used by the getCustomClassloader tests.

This function should be used in all of the tests that pass in directories and 
not paths that are to jar files.  Without it, NiFi builds that enable tests 
will fail on Mac OS X if .DS_Store files exist in the testing directory.

This problem also effects TestIdentifyMimeType.java, where it is doing a 
directory listing, and should ignore "." files.





  was:
DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in any 
directory on Mac OS X. 

Any call made to getURLsForClasspath  or getCustomClassLoader passing 
directories an not paths to jars for modules will return these files as urls 
unless they are specifically filtered out using the FileNameFilter parameter.

The FileNameFilter parameter to these functions can be used to protect against 
this, and be sure the URL's returned are for the expected items.  In fact there 
is a function  getJarFilenameFilter in the TestClassloaderUtils already, that 
is used by the getCustomClassloader tests.

This function should be used in all of the tests that pass in directories and 
not paths that are to jar files.  Without it, NiFi builds that enable tests 
will fail on Mac OS X if .DS_Store files exist in the testing directory.






> Tests in TestClassloaderUtils should consistently use the FileNameFilter when 
> using directories
> ---
>
> Key: NIFI-3454
> URL: https://issues.apache.org/jira/browse/NIFI-3454
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
> Environment: Mac OS X ( any version )  Java 1.8.0_31  Maven 3.3.9
>Reporter: Otto Fowler
>
> DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in 
> any directory on Mac OS X. 
> Any call made to getURLsForClasspath  or getCustomClassLoader passing 
> directories an not paths to jars for modules will return these files as urls 
> unless they are specifically filtered out using the FileNameFilter parameter.
> The FileNameFilter parameter to these functions can be used to protect 
> against this, and be sure the URL's returned are for the expected items.  In 
> fact there is a function  getJarFilenameFilter in the TestClassloaderUtils 
> already, that is used by the getCustomClassloader tests.
> This function should be used in all of the tests that pass in directories and 
> not paths that are to jar files.  Without it, NiFi builds that enable tests 
> will fail on Mac OS X if .DS_Store files exist in the testing directory.
> This problem also effects TestIdentifyMimeType.java, where it is doing a 
> directory listing, and should ignore "." files.
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (NIFI-3459) Weird blue highlighting of breadcrumbs

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan reassigned NIFI-3459:
-

Assignee: Scott Aslan

> Weird blue highlighting of breadcrumbs
> --
>
> Key: NIFI-3459
> URL: https://issues.apache.org/jira/browse/NIFI-3459
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Reporter: Scott Aslan
>Assignee: Scott Aslan
>Priority: Trivial
>
> When clicking on the breadcrumbs to navigate to a group there seems to be 
> some weird blue highlighting.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (NIFI-3458) Processor Fill Color Preview Bottom border missing.

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan reassigned NIFI-3458:
-

Assignee: Scott Aslan

> Processor Fill Color Preview Bottom border missing.
> ---
>
> Key: NIFI-3458
> URL: https://issues.apache.org/jira/browse/NIFI-3458
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Reporter: Scott Aslan
>Assignee: Scott Aslan
>Priority: Trivial
>
> For certain screen sizes the bottom border on the processor fill color 
> preview gets cut off.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (NIFI-3459) Weird blue highlighting of breadcrumbs

[Scott Aslan (JIRA)]

Scott Aslan created NIFI-3459:
-

 Summary: Weird blue highlighting of breadcrumbs
 Key: NIFI-3459
 URL: https://issues.apache.org/jira/browse/NIFI-3459
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core UI
Reporter: Scott Aslan
Priority: Trivial


When clicking on the breadcrumbs to navigate to a group there seems to be some 
weird blue highlighting.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (NIFI-3458) Processor Fill Color Preview Bottom border missing.

[Scott Aslan (JIRA)]

Scott Aslan created NIFI-3458:
-

 Summary: Processor Fill Color Preview Bottom border missing.
 Key: NIFI-3458
 URL: https://issues.apache.org/jira/browse/NIFI-3458
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core UI
Reporter: Scott Aslan
Priority: Trivial


For certain screen sizes the bottom border on the processor fill color preview 
gets cut off.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3450) The styles for the search icon in the template upload dialog are incorrect.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860262#comment-15860262
 ] 

ASF GitHub Bot commented on NIFI-3450:
--

GitHub user scottyaslan opened a pull request:

https://github.com/apache/nifi/pull/1495

[NIFI-3450] update upload template dialog browse button

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scottyaslan/nifi NIFI-3450

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1495.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1495


commit d1fbe7824cd689dd457019db403e2f752593880a
Author: Scott Aslan 
Date:   2017-02-09T16:29:49Z

[NIFI-3450] update upload template dialog browse button




> The styles for the search icon in the template upload dialog are incorrect.
> ---
>
> Key: NIFI-3450
> URL: https://issues.apache.org/jira/browse/NIFI-3450
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Reporter: Scott Aslan
>Assignee: Scott Aslan
>Priority: Trivial
>
> On the operate palette click to upload a template. Hover over the search 
> iconnotice it does not display its hover state. Also it looks like the 
> bottom border of the button for the search icon is cut off. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3450) The styles for the search icon in the template upload dialog are incorrect.

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan updated NIFI-3450:
--
Status: Patch Available  (was: In Progress)

> The styles for the search icon in the template upload dialog are incorrect.
> ---
>
> Key: NIFI-3450
> URL: https://issues.apache.org/jira/browse/NIFI-3450
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Reporter: Scott Aslan
>Assignee: Scott Aslan
>Priority: Trivial
>
> On the operate palette click to upload a template. Hover over the search 
> iconnotice it does not display its hover state. Also it looks like the 
> bottom border of the button for the search icon is cut off. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1495: [NIFI-3450] update upload template dialog browse bu...

[scottyaslan]

GitHub user scottyaslan opened a pull request:

https://github.com/apache/nifi/pull/1495

[NIFI-3450] update upload template dialog browse button

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scottyaslan/nifi NIFI-3450

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1495.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1495


commit d1fbe7824cd689dd457019db403e2f752593880a
Author: Scott Aslan 
Date:   2017-02-09T16:29:49Z

[NIFI-3450] update upload template dialog browse button




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1657) Provide support for various locales in CI builds

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860256#comment-15860256
 ] 

ASF GitHub Bot commented on NIFI-1657:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1480
  
@pvillard31 - this seems to be working now


> Provide support for various locales in CI builds
> 
>
> Key: NIFI-1657
> URL: https://issues.apache.org/jira/browse/NIFI-1657
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Tools and Build
>Reporter: Aldrin Piri
>Assignee: Andre
>Priority: Minor
>
> There were issues stemming from locale and building on more than the default 
> would help us catch similar issues moving forward.
> From https://github.com/apache/nifi/pull/292, this can be accomplished via 
> something like:
> -Duser.language=fr -Duser.region=FR



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1480: NIFI-1657 - Configure travis-ci to run mvn verify in jp an...

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1480
  
@pvillard31 - this seems to be working now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3449) Create Google Cloud Platform/Google Cloud Storage Processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860093#comment-15860093
 ] 

ASF GitHub Bot commented on NIFI-3449:
--

Github user gene-telligent commented on the issue:

https://github.com/apache/nifi/pull/1482
  
@fmaritato Hey frank! First of all, thank you so much for getting the ball 
started on the GCP processors -- I didn't mean for this to seem like a snub; I 
just had some spare time and pretty much did a 1:1 copy of the AWS processors 
(or at least their design principles). The biggest thing was adding those 
credential controllers, since it seems like that's the most appropriate pattern 
for dealing with credentials.

I've made the changes to allow for Project ID being dynamic. And 
unfortunately yes, for the time being, if you're using separate credentials for 
different GCP processors, you'll need separate credential services. That being 
said, if you're primarily using Application Default credentials, you can apply 
those to multiple processors without a problem. 


> Create Google Cloud Platform/Google Cloud Storage Processors
> 
>
> Key: NIFI-3449
> URL: https://issues.apache.org/jira/browse/NIFI-3449
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Gene Peters
>  Labels: features
>
> Hi all,
> We had a need in our production deployments to interact with Google Cloud 
> Storage. At the time, NIFI-2809 hadn't seen much movement, and after applying 
> the patch I found that the configuration was too specific for my needs (it's 
> hardcoded to use "Application Default" credentials, everything uploaded to 
> GCS is uploaded with the "public" ACL, etc). So I created a series of 
> Processors / Controller Services based off of the AWS NiFi library, and would 
> like to contribute them. 
> Features:
> * All credentialing is handled by a controller service, allowing multiple 
> processors to use the same service / credentials
> * An Abstract processor is provided which forms the basis for all GCP related 
> processors.
> * The standard Google Cloud Storage operations are supported, very similarly 
> to the AWS S3 processors: ListGCSBucket, DeleteGCSObject, FetchGCSObject, 
> PutGCSObject
> * Everything is documented and unit tested. 
> * I've also provided integration tests, but they're disabled by default (as 
> they require Google Cloud credentials). To run them, use the flag 
> {{skipGCPIntegrationTests=false}}
> Todo:
> * The GCP Java library's ReadChannel objects implement the "restorable" 
> interface, which allows for state saving / checkpointing. I'd really like to 
> leverage this with the State support that NiFi provides, but it would require 
> serializing / deserializing the object. 
> I'm going to be submitting this as a pull request through GitHub. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1482: NIFI-3449: Adding GCP Framework and GCS Processors

[gene-telligent]

Github user gene-telligent commented on the issue:

https://github.com/apache/nifi/pull/1482
  
@fmaritato Hey frank! First of all, thank you so much for getting the ball 
started on the GCP processors -- I didn't mean for this to seem like a snub; I 
just had some spare time and pretty much did a 1:1 copy of the AWS processors 
(or at least their design principles). The biggest thing was adding those 
credential controllers, since it seems like that's the most appropriate pattern 
for dealing with credentials.

I've made the changes to allow for Project ID being dynamic. And 
unfortunately yes, for the time being, if you're using separate credentials for 
different GCP processors, you'll need separate credential services. That being 
said, if you're primarily using Application Default credentials, you can apply 
those to multiple processors without a problem. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #42: MINIFI-186 Updating Copyright statements t...

[apiri]

GitHub user apiri reopened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/42

MINIFI-186 Updating Copyright statements to reflect the new year.

MINIFI-186 Updating Copyright statements to reflect the new year.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apiri/nifi-minifi-cpp minifi-186

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/42.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #42






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #42: MINIFI-186 Updating Copyright statements t...

[apiri]

Github user apiri closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/42


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3356) Provide a newly refactored provenance repository

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860027#comment-15860027
 ] 

ASF GitHub Bot commented on NIFI-3356:
--

Github user mosermw commented on the issue:

https://github.com/apache/nifi/pull/1493
  
Some neat stuff here, @markap14 .  I noticed you changed a couple of the 
same files as PR #1475.  Since #1475 is almost complete, it would be nice to 
commit that first.  Merging that into this PR shouldn't be bad at all, and if 
we're lucky it will still apply cleanly!


> Provide a newly refactored provenance repository
> 
>
> Key: NIFI-3356
> URL: https://issues.apache.org/jira/browse/NIFI-3356
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> The Persistent Provenance Repository has been redesigned a few different 
> times over several years. The original design for the repository was to 
> provide storage of events and sequential iteration over those events via a 
> Reporting Task. After that, we added the ability to compress the data so that 
> it could be held longer. We then introduced the notion of indexing and 
> searching via Lucene. We've since made several more modifications to try to 
> boost performance.
> At this point, however, the repository is still the bottleneck for many flows 
> that handle large volumes of small FlowFiles. We need a new implementation 
> that is based around the current goals for the repository and that can 
> provide better throughput.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1493: NIFI-3356: Initial implementation of writeahead provenance...

[mosermw]

Github user mosermw commented on the issue:

https://github.com/apache/nifi/pull/1493
  
Some neat stuff here, @markap14 .  I noticed you changed a couple of the 
same files as PR #1475.  Since #1475 is almost complete, it would be nice to 
commit that first.  Merging that into this PR shouldn't be bad at all, and if 
we're lucky it will still apply cleanly!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-3457) ConvertAvroSchema should not require InputSchema

[Matt Burgess (JIRA)]

Matt Burgess created NIFI-3457:
--

 Summary: ConvertAvroSchema should not require InputSchema
 Key: NIFI-3457
 URL: https://issues.apache.org/jira/browse/NIFI-3457
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Reporter: Matt Burgess


Currently, the ConvertAvroSchema processor requires both an Input and Output 
Schema in order to perform a conversion.  The Input Schema should only be 
necessary if the Avro file(s) coming in are "raw" (aka schema-less). Otherwise 
the schema should be in the file and if the Input Schema property is left 
blank, the processor should attempt to retrieve the schema from the incoming 
Avro file. If the incoming file is schema-less and the property is left blank, 
then an error bulletin can be issued, saying that no schema was found or 
supplied.

The proposed changes are these:

1) Make the Input Schema property .required(false)
2) If the Input Schema property is empty, attempt to retrieve the schema from 
the incoming flow file. If no schema exists, issue an error
3) Update the processor documentation to reflect this additional behavior.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3403) NPE in InvokeHTTP

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-3403:
-
 Assignee: Pierre Villard
Fix Version/s: 1.2.0
   Status: Patch Available  (was: Open)

> NPE in InvokeHTTP
> -
>
> Key: NIFI-3403
> URL: https://issues.apache.org/jira/browse/NIFI-3403
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.1.1, 0.7.1
>Reporter: Brandon DeVries
>Assignee: Pierre Villard
>Priority: Minor
> Fix For: 1.2.0
>
>
> InvokeHTTP throws an NPE when running in "source" mode if given a value for 
> "Attributes to Send".  If this value is set, it attempts to do\[1\]...
> {code}
> requestFlowFile.getAttributes(){code}
> ...without checking if requestFlowFile == null (which will be the case when 
> running in "source" mode).
> \[1\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L859



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3403) NPE in InvokeHTTP

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860018#comment-15860018
 ] 

ASF GitHub Bot commented on NIFI-3403:
--

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1494

NIFI-3403 - Fix NPE in InvokeHTTP

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [X] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-3403

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1494.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1494


commit 37496d336c47f8c9a542f02e47314ace07b43528
Author: Pierre Villard 
Date:   2017-02-09T19:10:33Z

NIFI-3403 - Fix NPE in InvokeHTTP




> NPE in InvokeHTTP
> -
>
> Key: NIFI-3403
> URL: https://issues.apache.org/jira/browse/NIFI-3403
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Brandon DeVries
>Priority: Minor
>
> InvokeHTTP throws an NPE when running in "source" mode if given a value for 
> "Attributes to Send".  If this value is set, it attempts to do\[1\]...
> {code}
> requestFlowFile.getAttributes(){code}
> ...without checking if requestFlowFile == null (which will be the case when 
> running in "source" mode).
> \[1\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L859



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1494: NIFI-3403 - Fix NPE in InvokeHTTP

[pvillard31]

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1494

NIFI-3403 - Fix NPE in InvokeHTTP

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [X] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-3403

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1494.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1494


commit 37496d336c47f8c9a542f02e47314ace07b43528
Author: Pierre Villard 
Date:   2017-02-09T19:10:33Z

NIFI-3403 - Fix NPE in InvokeHTTP




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #41: MINIFI-184: Add Security Support

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/41


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3356) Provide a newly refactored provenance repository

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-3356:
-
Status: Patch Available  (was: In Progress)

> Provide a newly refactored provenance repository
> 
>
> Key: NIFI-3356
> URL: https://issues.apache.org/jira/browse/NIFI-3356
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> The Persistent Provenance Repository has been redesigned a few different 
> times over several years. The original design for the repository was to 
> provide storage of events and sequential iteration over those events via a 
> Reporting Task. After that, we added the ability to compress the data so that 
> it could be held longer. We then introduced the notion of indexing and 
> searching via Lucene. We've since made several more modifications to try to 
> boost performance.
> At this point, however, the repository is still the bottleneck for many flows 
> that handle large volumes of small FlowFiles. We need a new implementation 
> that is based around the current goals for the repository and that can 
> provide better throughput.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3356) Provide a newly refactored provenance repository

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1585#comment-1585
 ] 

ASF GitHub Bot commented on NIFI-3356:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/1493

NIFI-3356: Initial implementation of writeahead provenance repository

- The idea behind NIFI-3356 was to improve the efficiency and throughput of 
the Provenance Repository, as it is often the bottleneck. While testing the 
newly designed repository,
  a handful of other, fairly minor, changes were made to improve efficiency 
as well, as these came to light when testing the new repository:

- Use a BufferedOutputStream within StandardProcessSession (via a 
ClaimCache abstraction) in order to avoid continually writing to 
FileOutputStream when writing many small FlowFiles
- Updated threading model of MinimalLockingWriteAheadLog - now performs 
serialization outside of lock and writes to a 'synchronized' OutputStream
- Change minimum scheduling period for components from 30 microseconds to 1 
nanosecond. ScheduledExecutor is very inconsistent with timing of task 
scheduling. With the bored.yield.duration
  now present, this value doesn't need to be set to 30 microseconds. This 
was originally done to avoid processors that had no work from dominating the 
CPU. However, now that we will yield
  when processors have no work, this results in slowing down processors 
that are able to perform work.
- Allow nifi.properties to specify multiple directories for FlowFile 
Repository
- If backpressure is engaged while running a batch of sessions, then stop 
batch processing earlier. This helps FlowFiles to move through the system much 
more smoothly instead of the
  herky-jerky queuing that we previously saw at very high rates of 
FlowFiles.
- Added NiFi PID to log message when starting nifi. This was simply an 
update to the log message that provides helpful information.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-3356

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1493.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1493


commit babbe4e42a6cc2b0d124691bb398f1ec33d3b8c8
Author: Mark Payne 
Date:   2016-12-09T15:52:33Z

NIFI-3356: Initial implementation of writeahead provenance repository
- The idea behind NIFI-3356 was to improve the efficiency and throughput of 
the Provenance Repository, as it is often the bottleneck. While testing the 
newly designed repository,
  a handful of other, fairly minor, changes were made to improve efficiency 
as well, as these came to light when testing the new repository:

- Use a BufferedOutputStream within StandardProcessSession (via a 
ClaimCache abstraction) in order to avoid continually writing to 
FileOutputStream when writing many small FlowFiles
- Updated 

[GitHub] nifi pull request #1493: NIFI-3356: Initial implementation of writeahead pro...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/1493

NIFI-3356: Initial implementation of writeahead provenance repository

- The idea behind NIFI-3356 was to improve the efficiency and throughput of 
the Provenance Repository, as it is often the bottleneck. While testing the 
newly designed repository,
  a handful of other, fairly minor, changes were made to improve efficiency 
as well, as these came to light when testing the new repository:

- Use a BufferedOutputStream within StandardProcessSession (via a 
ClaimCache abstraction) in order to avoid continually writing to 
FileOutputStream when writing many small FlowFiles
- Updated threading model of MinimalLockingWriteAheadLog - now performs 
serialization outside of lock and writes to a 'synchronized' OutputStream
- Change minimum scheduling period for components from 30 microseconds to 1 
nanosecond. ScheduledExecutor is very inconsistent with timing of task 
scheduling. With the bored.yield.duration
  now present, this value doesn't need to be set to 30 microseconds. This 
was originally done to avoid processors that had no work from dominating the 
CPU. However, now that we will yield
  when processors have no work, this results in slowing down processors 
that are able to perform work.
- Allow nifi.properties to specify multiple directories for FlowFile 
Repository
- If backpressure is engaged while running a batch of sessions, then stop 
batch processing earlier. This helps FlowFiles to move through the system much 
more smoothly instead of the
  herky-jerky queuing that we previously saw at very high rates of 
FlowFiles.
- Added NiFi PID to log message when starting nifi. This was simply an 
update to the log message that provides helpful information.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-3356

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1493.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1493


commit babbe4e42a6cc2b0d124691bb398f1ec33d3b8c8
Author: Mark Payne 
Date:   2016-12-09T15:52:33Z

NIFI-3356: Initial implementation of writeahead provenance repository
- The idea behind NIFI-3356 was to improve the efficiency and throughput of 
the Provenance Repository, as it is often the bottleneck. While testing the 
newly designed repository,
  a handful of other, fairly minor, changes were made to improve efficiency 
as well, as these came to light when testing the new repository:

- Use a BufferedOutputStream within StandardProcessSession (via a 
ClaimCache abstraction) in order to avoid continually writing to 
FileOutputStream when writing many small FlowFiles
- Updated threading model of MinimalLockingWriteAheadLog - now performs 
serialization outside of lock and writes to a 'synchronized' OutputStream
- Change minimum scheduling period for components from 30 microseconds to 1 
nanosecond. 

[GitHub] nifi pull request #1492: NIFI-3405 - Add uptime to JVM section in System Dia...

[pvillard31]

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1492

NIFI-3405 - Add uptime to JVM section in System Diagnostics

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-3405

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1492.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1492


commit 93696e0995d8723a3e028001e832f6cc0a395f6c
Author: Pierre Villard 
Date:   2017-02-09T17:59:53Z

NIFI-3405 - Add uptime to JVM section in System Diagnostics




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3405) Add uptime to JVM section in System Diagnostics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859938#comment-15859938
 ] 

ASF GitHub Bot commented on NIFI-3405:
--

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1492

NIFI-3405 - Add uptime to JVM section in System Diagnostics

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-3405

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1492.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1492


commit 93696e0995d8723a3e028001e832f6cc0a395f6c
Author: Pierre Villard 
Date:   2017-02-09T17:59:53Z

NIFI-3405 - Add uptime to JVM section in System Diagnostics




> Add uptime to JVM section in System Diagnostics
> ---
>
> Key: NIFI-3405
> URL: https://issues.apache.org/jira/browse/NIFI-3405
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Aldrin Piri
> Fix For: 1.2.0
>
>
> When diagnosing system performance, it would be helpful to see how long the 
> current JVM has been alive.  In one case, this could provide context helpful 
> for interpreting garbage collection.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3405) Add uptime to JVM section in System Diagnostics

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-3405:
-
 Assignee: Pierre Villard
Fix Version/s: 1.2.0
   Status: Patch Available  (was: Open)

> Add uptime to JVM section in System Diagnostics
> ---
>
> Key: NIFI-3405
> URL: https://issues.apache.org/jira/browse/NIFI-3405
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Aldrin Piri
>Assignee: Pierre Villard
> Fix For: 1.2.0
>
>
> When diagnosing system performance, it would be helpful to see how long the 
> current JVM has been alive.  In one case, this could provide context helpful 
> for interpreting garbage collection.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi-minifi-cpp issue #41: MINIFI-184: Add Security Support

[apiri]

Github user apiri commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/41
  
Changes look good.  I saw there were a couple of inconsistencies with some 
of the docs and typos but corrected those and will incorporate in the merge.  
Thanks for the fixes!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-3455) GenerateTableFetch fails for very large row counts

[Peter Wicks (JIRA)]

Peter Wicks created NIFI-3455:
-

 Summary: GenerateTableFetch fails for very large row counts
 Key: NIFI-3455
 URL: https://issues.apache.org/jira/browse/NIFI-3455
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.2.0
Reporter: Peter Wicks
Assignee: Peter Wicks
Priority: Minor
 Fix For: 1.2.0


When retrieving the row count for the total number of rows to page over 
GenerateTableFetch gets the row count as an integer. In some cases the value 
returned is greater then the maximum value of Int32 causing a looping of the 
value, resulting in a negative number.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (NIFI-3450) The styles for the search icon in the template upload dialog are incorrect.

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan reassigned NIFI-3450:
-

Assignee: Scott Aslan

> The styles for the search icon in the template upload dialog are incorrect.
> ---
>
> Key: NIFI-3450
> URL: https://issues.apache.org/jira/browse/NIFI-3450
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Reporter: Scott Aslan
>Assignee: Scott Aslan
>Priority: Trivial
>
> On the operate palette click to upload a template. Hover over the search 
> iconnotice it does not display its hover state. Also it looks like the 
> bottom border of the button for the search icon is cut off. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi-minifi-cpp issue #47: MINIFI-190: Creating initial commit with addition...

[phrocker]

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/47
  
I have found that the flow repository isn't cleaning up after itself. Will 
resolve that first. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3360) Modularize canvas directory

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859650#comment-15859650
 ] 

ASF GitHub Bot commented on NIFI-3360:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1487
  
Will review...


> Modularize canvas directory
> ---
>
> Key: NIFI-3360
> URL: https://issues.apache.org/jira/browse/NIFI-3360
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Scott Aslan
>Assignee: Scott Aslan
>
> Need to modularize (re-package) UI components to promote maintainability and 
> eventual reusability for UI extensions. Components are already conceptually 
> modular however circular references exists which should not be allowed. Need 
> to re-package functions to eliminate circular
> references.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1487: [NIFI-3360] Modularize Canvas

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1487
  
Will review...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #47: MINIFI-190: Creating initial commit with addition...

[phrocker]

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/47
  
I'll be happy to break this apart


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #47: MINIFI-190: Creating initial commit with a...

[phrocker]

GitHub user phrocker opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/47

MINIFI-190: Creating initial commit with additional tests

Additional tests will be updated in separate tickets as needed.

Resolves MINIFI-194 by using a semaphore in place of the
FlowController instance. Stop is performed outside of the
signal handler to avoid synchronicity issues.

Resolves MINIFI-192 by using lock_guard based on a conditional

Resolves issues found with MINIFI-190 regarding GetFile. Added
pragma definitions for GCC < 4.9

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFI-190

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/47.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #47


commit e3ed8da1739288774a018c8b2422073802362014
Author: Marc Parisi 
Date:   2017-02-03T14:23:42Z

MINIFI-190: Creating initial commit with changes found

Resolves MINIFI-194 by using a semaphore in place of the
FlowController instance. Stop is performed outside of the
signal handler to avoid synchronicity issues.

Resolves MINIFI-192 by using lock_guard based on a conditional

Resolves issues found with MINIFI-190 regarding GetFile. Added
pragma definitions for GCC < 4.9




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (NIFI-3438) Ability to link to a view in Nifi (process group, processor, etc...)

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim resolved NIFI-3438.
--
Resolution: Duplicate

Thanks [~ncarenza].  Closing as dupe.

> Ability to link to a view in Nifi (process group, processor, etc...)
> 
>
> Key: NIFI-3438
> URL: https://issues.apache.org/jira/browse/NIFI-3438
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Nicholas Carenza
>Priority: Minor
>
> I would like to be able to share a link to a particular view in the Nifi UI. 
> For example:
> #/ might just bring me to that processor, center it in view and 
> reset the zoom. If it is a process group, it might go to it an bring all 
> processors into view.
> #//config might then also bring up the configuration for that 
> processor
> #//config/properties might then also set the tab to 'properties'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3454) Tests in TestClassloaderUtils should consistently use the FileNameFilter when using directories

[Otto Fowler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler updated NIFI-3454:
--
Description: 
DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in any 
directory on Mac OS X. 

Any call made to getURLsForClasspath  or getCustomClassLoader passing 
directories an not paths to jars for modules will return these files as urls 
unless they are specifically filtered out using the FileNameFilter parameter.

The FileNameFilter parameter to these functions can be used to protect against 
this, and be sure the URL's returned are for the expected items.  In fact there 
is a function  getJarFilenameFilter in the TestClassloaderUtils already, that 
is used by the getCustomClassloader tests.

This function should be used in all of the tests that pass in directories and 
not paths that are to jar files.  Without it, NiFi builds that enable tests 
will fail on Mac OS X if .DS_Store files exist in the testing directory.





  was:
DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in any 
directory on Mac OS X. 

Any call made to getURLsForClasspath  or getCustomClassLoader passing 
directories an not jars for modules will return these files as urls unless they 
are specifically filtered out using the FileNameFilter parameter.

The FileNameFilter parameter to these functions can be used to protect against 
this, and be sure the URL's returned are for the expected items.  In fact there 
is a function  getJarFilenameFilter in the TestClassloaderUtils already, that 
is used by the getCustomClassloader tests.

This function should be used in all of the tests that pass in directories and 
not paths that are to jar files.  Without it, NiFi builds that enable tests 
will fail on Mac OS X if .DS_Store files exist in the testing directory.






> Tests in TestClassloaderUtils should consistently use the FileNameFilter when 
> using directories
> ---
>
> Key: NIFI-3454
> URL: https://issues.apache.org/jira/browse/NIFI-3454
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
> Environment: Mac OS X ( any version )  Java 1.8.0_31  Maven 3.3.9
>Reporter: Otto Fowler
>
> DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in 
> any directory on Mac OS X. 
> Any call made to getURLsForClasspath  or getCustomClassLoader passing 
> directories an not paths to jars for modules will return these files as urls 
> unless they are specifically filtered out using the FileNameFilter parameter.
> The FileNameFilter parameter to these functions can be used to protect 
> against this, and be sure the URL's returned are for the expected items.  In 
> fact there is a function  getJarFilenameFilter in the TestClassloaderUtils 
> already, that is used by the getCustomClassloader tests.
> This function should be used in all of the tests that pass in directories and 
> not paths that are to jar files.  Without it, NiFi builds that enable tests 
> will fail on Mac OS X if .DS_Store files exist in the testing directory.
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3454) Tests in TestClassloaderUtils should consistently use the FileNameFilter when using directories

[Otto Fowler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler updated NIFI-3454:
--
Summary: Tests in TestClassloaderUtils should consistently use the 
FileNameFilter when using directories  (was: Tests in TestClassloaderUtils 
should consistently use the FileNameFilter)

> Tests in TestClassloaderUtils should consistently use the FileNameFilter when 
> using directories
> ---
>
> Key: NIFI-3454
> URL: https://issues.apache.org/jira/browse/NIFI-3454
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
> Environment: Mac OS X ( any version )  Java 1.8.0_31  Maven 3.3.9
>Reporter: Otto Fowler
>
> DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in 
> any directory on Mac OS X. 
> Any call made to getURLsForClasspath  or getCustomClassLoader passing 
> directories an not jars for modules will return these files as urls unless 
> they are specifically filtered out using the FileNameFilter parameter.
> The FileNameFilter parameter to these functions can be used to protect 
> against this, and be sure the URL's returned are for the expected items.  In 
> fact there is a function  getJarFilenameFilter in the TestClassloaderUtils 
> already, that is used by the getCustomClassloader tests.
> This function should be used in all of the tests that pass in directories and 
> not paths that are to jar files.  Without it, NiFi builds that enable tests 
> will fail on Mac OS X if .DS_Store files exist in the testing directory.
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3454) Tests in TestClassloaderUtils should consistently use the FileNameFilter

[Otto Fowler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler updated NIFI-3454:
--
Description: 
DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in any 
directory on Mac OS X. 

Any call made to getURLsForClasspath  or getCustomClassLoader passing 
directories an not jars for modules will return these files as urls unless they 
are specifically filtered out using the FileNameFilter parameter.

The FileNameFilter parameter to these functions can be used to protect against 
this, and be sure the URL's returned are for the expected items.  In fact there 
is a function  getJarFilenameFilter in the TestClassloaderUtils already, that 
is used by the getCustomClassloader tests.

This function should be used in all of the tests that pass in directories and 
not paths that are to jar files.  Without it, NiFi builds that enable tests 
will fail on Mac OS X if .DS_Store files exist in the testing directory.





  was:
DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in any 
directory on Mac OS X. 

Any call made to getURLsForClasspath  or getCustomClassLoader passing 
directories an not jars for modules will return these files as urls unless they 
are specifically filtered out using the FileNameFilter parameter.

The FileNameFilter parameter to these functions can be used to protect against 
this, and be sure the URL's returned are for the expected items.  In fact there 
is a function  getJarFilenameFilter in the TestClassloaderUtils already, that 
is used by the getCustomClassloader tests.

This function should be used in all of the tests.  Without it, NiFi builds that 
enable tests will fail on Mac OS X if .DS_Store files exist in the testing 
directory.






> Tests in TestClassloaderUtils should consistently use the FileNameFilter
> 
>
> Key: NIFI-3454
> URL: https://issues.apache.org/jira/browse/NIFI-3454
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
> Environment: Mac OS X ( any version )  Java 1.8.0_31  Maven 3.3.9
>Reporter: Otto Fowler
>
> DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in 
> any directory on Mac OS X. 
> Any call made to getURLsForClasspath  or getCustomClassLoader passing 
> directories an not jars for modules will return these files as urls unless 
> they are specifically filtered out using the FileNameFilter parameter.
> The FileNameFilter parameter to these functions can be used to protect 
> against this, and be sure the URL's returned are for the expected items.  In 
> fact there is a function  getJarFilenameFilter in the TestClassloaderUtils 
> already, that is used by the getCustomClassloader tests.
> This function should be used in all of the tests that pass in directories and 
> not paths that are to jar files.  Without it, NiFi builds that enable tests 
> will fail on Mac OS X if .DS_Store files exist in the testing directory.
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3454) Tests in TestClassloaderUtils should consistently use the FileNameFilter

[Otto Fowler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler updated NIFI-3454:
--
Summary: Tests in TestClassloaderUtils should consistently use the 
FileNameFilter  (was: testGetURLsForClasspathWithDirectory fails on Mac OS X if 
.DS_Store files are present)

> Tests in TestClassloaderUtils should consistently use the FileNameFilter
> 
>
> Key: NIFI-3454
> URL: https://issues.apache.org/jira/browse/NIFI-3454
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
> Environment: Mac OS X ( any version )  Java 1.8.0_31  Maven 3.3.9
>Reporter: Otto Fowler
>
> DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in 
> any directory on Mac OS X. 
> Any call made to getURLsForClasspath  or getCustomClassLoader passing 
> directories an not jars for modules will return these files as urls unless 
> they are specifically filtered out using the FileNameFilter parameter.
> The FileNameFilter parameter to these functions can be used to protect 
> against this, and be sure the URL's returned are for the expected items.  In 
> fact there is a function  getJarFilenameFilter in the TestClassloaderUtils 
> already, that is used by the getCustomClassloader tests.
> This function should be used in all of the tests.  Without it, NiFi builds 
> that enable tests will fail on Mac OS X if .DS_Store files exist in the 
> testing directory.
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3454) testGetURLsForClasspathWithDirectory fails on Mac OS X if .DS_Store files are present

[Otto Fowler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler updated NIFI-3454:
--
Description: 
DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in any 
directory on Mac OS X. 

Any call made to getURLsForClasspath  or getCustomClassLoader passing 
directories an not jars for modules will return these files as urls unless they 
are specifically filtered out using the FileNameFilter parameter.

The FileNameFilter parameter to these functions can be used to protect against 
this, and be sure the URL's returned are for the expected items.  In fact there 
is a function  getJarFilenameFilter in the TestClassloaderUtils already, that 
is used by the getCustomClassloader tests.

This function should be used in all of the tests.  Without it, NiFi builds that 
enable tests will fail on Mac OS X if .DS_Store files exist in the testing 
directory.





  was:
testGetURLsForClasspathWithDirectory

DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in any 
directory on Mac OS X. 

Any call made to getURLsForClasspath will return these files as urls unless 
they are specifically filtered out using the FileNameFilter parameter.

testGetURLsForClasspathWithDirectory does not pass in a filter and thus fails 
it's assertion with the wrong number of urls returned.  This results in a 
failure of Nifi to build with tests enabled.

The fix for this is to pass a filter from the test function which will limit 
the urls to not null names, and names that end in .jar.

This is the approach taken in other areas of the code such as 
DBCPConnectionPool and JoltTransformJSON


> testGetURLsForClasspathWithDirectory fails on Mac OS X if .DS_Store files are 
> present
> -
>
> Key: NIFI-3454
> URL: https://issues.apache.org/jira/browse/NIFI-3454
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
> Environment: Mac OS X ( any version )  Java 1.8.0_31  Maven 3.3.9
>Reporter: Otto Fowler
>
> DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in 
> any directory on Mac OS X. 
> Any call made to getURLsForClasspath  or getCustomClassLoader passing 
> directories an not jars for modules will return these files as urls unless 
> they are specifically filtered out using the FileNameFilter parameter.
> The FileNameFilter parameter to these functions can be used to protect 
> against this, and be sure the URL's returned are for the expected items.  In 
> fact there is a function  getJarFilenameFilter in the TestClassloaderUtils 
> already, that is used by the getCustomClassloader tests.
> This function should be used in all of the tests.  Without it, NiFi builds 
> that enable tests will fail on Mac OS X if .DS_Store files exist in the 
> testing directory.
> 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3291) UI - Upgrade jQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859600#comment-15859600
 ] 

ASF GitHub Bot commented on NIFI-3291:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1486


> UI - Upgrade jQuery
> ---
>
> Key: NIFI-3291
> URL: https://issues.apache.org/jira/browse/NIFI-3291
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
> Fix For: 1.2.0
>
>
> Also consider upgrading all jQuery plugins too.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1486: NIFI-3291: Addressing more issues following jQuery upgrade

[scottyaslan]

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/1486
  
Thanks @mcgilman this has been merged to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3291) UI - Upgrade jQuery

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859598#comment-15859598
 ] 

ASF subversion and git services commented on NIFI-3291:
---

Commit bc7b17a64c8e995952fb6579ec577167453022ef in nifi's branch 
refs/heads/master from [~mcgilman]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=bc7b17a ]

NIFI-3291:
- Fixing slider issue when there is only one event in the lineage graph.
- Fixing scollbar issue when accessing the queue listing.


> UI - Upgrade jQuery
> ---
>
> Key: NIFI-3291
> URL: https://issues.apache.org/jira/browse/NIFI-3291
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
> Fix For: 1.2.0
>
>
> Also consider upgrading all jQuery plugins too.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (NIFI-3291) UI - Upgrade jQuery

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan resolved NIFI-3291.
---
Resolution: Fixed

> UI - Upgrade jQuery
> ---
>
> Key: NIFI-3291
> URL: https://issues.apache.org/jira/browse/NIFI-3291
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
> Fix For: 1.2.0
>
>
> Also consider upgrading all jQuery plugins too.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3291) UI - Upgrade jQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859601#comment-15859601
 ] 

ASF GitHub Bot commented on NIFI-3291:
--

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/1486
  
Thanks @mcgilman this has been merged to master.


> UI - Upgrade jQuery
> ---
>
> Key: NIFI-3291
> URL: https://issues.apache.org/jira/browse/NIFI-3291
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
> Fix For: 1.2.0
>
>
> Also consider upgrading all jQuery plugins too.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3291) UI - Upgrade jQuery

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859599#comment-15859599
 ] 

ASF subversion and git services commented on NIFI-3291:
---

Commit 74ecc20f00fe140e89d42fcc47b33000bb818c80 in nifi's branch 
refs/heads/master from [~mcgilman]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=74ecc20 ]

NIFI-3291:
- Ensuring slider step is non negative.

This closes #1486


> UI - Upgrade jQuery
> ---
>
> Key: NIFI-3291
> URL: https://issues.apache.org/jira/browse/NIFI-3291
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
> Fix For: 1.2.0
>
>
> Also consider upgrading all jQuery plugins too.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1486: NIFI-3291: Addressing more issues following jQuery ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1486


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1491: NIFI-3331 - TLS Toolkit - add the possibility to de...

[pvillard31]

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1491

NIFI-3331 - TLS Toolkit - add the possibility to define SAN in issued…

… certificates

=

Issue behind this improvement is: if you have a load balancer in front of 
your NiFi cluster, you may have the following issue - when accessing the UI 
(secured cluster, HTTPS) you might face a certificate issue because the 
certificate issuer won't match the address of the load balancer. One option to 
solve this issue is to add, in addition to the node host name, the load 
balancer address as Subject Alternative Names in the certificate. This should 
help when deploying tools like Knox, HAProxy, etc, in front of NiFi with HTTPS 
access.

This PR adds an option in the TLS Toolkit when using it in client mode. The 
option allows user to define a comma separated list of domains to add as SAN 
(in the example: the node and the load balancer). The domains will be added in 
the CSR sent to the CA and the CA will issue a signed certificate containing 
the same SAN.

=

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-3331

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1491.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1491


commit 1a1f8dedc40ea2769eb67c1fdaac06cb2e13e449
Author: Pierre Villard 
Date:   2017-02-09T13:47:39Z

NIFI-3331 - TLS Toolkit - add the possibility to define SAN in issued 
certificates




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3331) TLS Toolkit - add the possibility to define a SAN in issued certificates

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-3331:
-
   Labels: tls-toolkit  (was: )
Fix Version/s: 1.2.0
   Status: Patch Available  (was: Open)

> TLS Toolkit - add the possibility to define a SAN in issued certificates
> 
>
> Key: NIFI-3331
> URL: https://issues.apache.org/jira/browse/NIFI-3331
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>  Labels: tls-toolkit
> Fix For: 1.2.0
>
>
> To ease the deployment of a load balancer in front of NiFi, it would be nice 
> to allow users to define a SAN in certificates issued by the CA.
> To load balance the access to the UI or even with a ListenHTTP processor, 
> both will cause errors with a "Host mismatch" kind of error because of 
> different fqdn between nodes certificate and LB certificate. This is also 
> discussed here: http://stackoverflow.com/questions/40035356/nifi-load-balancer



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3331) TLS Toolkit - add the possibility to define a SAN in issued certificates

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859589#comment-15859589
 ] 

ASF GitHub Bot commented on NIFI-3331:
--

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1491

NIFI-3331 - TLS Toolkit - add the possibility to define SAN in issued…

… certificates

=

Issue behind this improvement is: if you have a load balancer in front of 
your NiFi cluster, you may have the following issue - when accessing the UI 
(secured cluster, HTTPS) you might face a certificate issue because the 
certificate issuer won't match the address of the load balancer. One option to 
solve this issue is to add, in addition to the node host name, the load 
balancer address as Subject Alternative Names in the certificate. This should 
help when deploying tools like Knox, HAProxy, etc, in front of NiFi with HTTPS 
access.

This PR adds an option in the TLS Toolkit when using it in client mode. The 
option allows user to define a comma separated list of domains to add as SAN 
(in the example: the node and the load balancer). The domains will be added in 
the CSR sent to the CA and the CA will issue a signed certificate containing 
the same SAN.

=

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-3331

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1491.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1491


commit 1a1f8dedc40ea2769eb67c1fdaac06cb2e13e449
Author: Pierre Villard 
Date:   2017-02-09T13:47:39Z

NIFI-3331 - TLS Toolkit - add the possibility to define SAN in issued 
certificates




> TLS Toolkit - add the possibility to define a SAN in issued certificates
> 
>
> Key: NIFI-3331
> URL: https://issues.apache.org/jira/browse/NIFI-3331
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> To ease the deployment of a load balancer in front of NiFi, it would be nice 
> to allow users to define a SAN in certificates issued by the CA.
> To load balance the access to the UI or even with a ListenHTTP processor, 
> both will cause errors with a "Host mismatch" kind of error because of 
> different fqdn between nodes certificate and LB certificate. This is also 
> discussed here: http://stackoverflow.com/questions/40035356/nifi-load-balancer



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (NIFI-3454) testGetURLsForClasspathWithDirectory fails on Mac OS X if .DS_Store files are present

[Otto Fowler (JIRA)]

Otto Fowler created NIFI-3454:
-

 Summary: testGetURLsForClasspathWithDirectory fails on Mac OS X if 
.DS_Store files are present
 Key: NIFI-3454
 URL: https://issues.apache.org/jira/browse/NIFI-3454
 Project: Apache NiFi
  Issue Type: Bug
  Components: Tools and Build
Affects Versions: 1.1.1
 Environment: Mac OS X ( any version )  Java 1.8.0_31  Maven 3.3.9
Reporter: Otto Fowler


testGetURLsForClasspathWithDirectory

DS_Store files [https://en.wikipedia.org/wiki/.DS_Store]  may be present in any 
directory on Mac OS X. 

Any call made to getURLsForClasspath will return these files as urls unless 
they are specifically filtered out using the FileNameFilter parameter.

testGetURLsForClasspathWithDirectory does not pass in a filter and thus fails 
it's assertion with the wrong number of urls returned.  This results in a 
failure of Nifi to build with tests enabled.

The fix for this is to pass a filter from the test function which will limit 
the urls to not null names, and names that end in .jar.

This is the approach taken in other areas of the code such as 
DBCPConnectionPool and JoltTransformJSON



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (NIFI-3331) TLS Toolkit - add the possibility to define a SAN in issued certificates

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard reassigned NIFI-3331:


Assignee: Pierre Villard

> TLS Toolkit - add the possibility to define a SAN in issued certificates
> 
>
> Key: NIFI-3331
> URL: https://issues.apache.org/jira/browse/NIFI-3331
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> To ease the deployment of a load balancer in front of NiFi, it would be nice 
> to allow users to define a SAN in certificates issued by the CA.
> To load balance the access to the UI or even with a ListenHTTP processor, 
> both will cause errors with a "Host mismatch" kind of error because of 
> different fqdn between nodes certificate and LB certificate. This is also 
> discussed here: http://stackoverflow.com/questions/40035356/nifi-load-balancer



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (NIFI-3453) Provenance Events dont have the "relationship" from where the event came

[Scott Reisdorf (JIRA)]

Scott Reisdorf created NIFI-3453:


 Summary: Provenance Events dont have the "relationship" from where 
the event came
 Key: NIFI-3453
 URL: https://issues.apache.org/jira/browse/NIFI-3453
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.0.1
Reporter: Scott Reisdorf
 Attachments: Screen Shot 2017-02-09 at 5.36.08 AM.png

I have a reporting task that processes the Provenance Events in NiFi.
I would like to know the relationship the event came from; however, that field 
doesn't seem to be getting populated. (screenshots attached).

No matter what processor or flow I create the "relationship" field is always 
null.

I am running with NiFi 1.0.1,









--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1490: NIFI-3452: Add Wait processor Wait Mode property

[ijokarumawak]

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/1490
  
Dear reviewer, 
an example flow template is available here:
https://gist.github.com/ijokarumawak/85a3d77297ea94614e9f3f2a9dabca67


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3452) Add Wait processor Wait Mode property

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859459#comment-15859459
 ] 

ASF GitHub Bot commented on NIFI-3452:
--

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/1490
  
Dear reviewer, 
an example flow template is available here:
https://gist.github.com/ijokarumawak/85a3d77297ea94614e9f3f2a9dabca67


> Add Wait processor Wait Mode property
> -
>
> Key: NIFI-3452
> URL: https://issues.apache.org/jira/browse/NIFI-3452
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Attachments: wait-for-a-par-of-flow-to-finish.png
>
>
> NiFi back pressure is handled per relationship and as long as a relationship 
> has room to receive more flow files, source processor is scheduled to run.
> However, this behavior is not ideal in some cases. For example, when there is 
> very computationally expensive task and user wants to limit the number of 
> FlowFiles can be processed at a given time, it's not always possible to limit 
> the rate by existing RateControl nor back-pressure mechanism.
> As a more practical example, in the following flow, it's expected the GetSQS 
> is triggered only when the previous FlowFile has been processed completely.
> Node 1 is parsing a flow file (indicated by the X in the connection between 
> FetchS3Object and Parse). Both connections have a back-pressure threshold of 
> 1, but because the object is already fetched, the first connection is empty 
> and can thus be filled. This means that, if a new item becomes available in 
> the queue, both of the following cases can happen with equal probability:
> {code}
> Case 1:
> --  -  -
> Node 1: | GetSQS | -X-> | FetchS3Object | -X-> | Parse |
> --  -  -
> --  -  -
> Node 2: | GetSQS | ---> | FetchS3Object | ---> | Parse |
> --  -  -
> Case 2:
> --  -  -
> Node 1: | GetSQS | ---> | FetchS3Object | -X-> | Parse |
> --  -  -
> --  -  -
> Node 2: | GetSQS | -X-> | FetchS3Object | ---> | Parse |
> --  -  -
> {code}
> To achieve that, we could improve Wait processor as follows.
> NiFi scheduler checks downstream relationship availability, when it's full, 
> the processor won't be scheduled to run. In case a source processor has 
> multiple outgoing relationships, and if ANY of those is full, the processor 
> won't be scheduled.
> (This is how processor scheduling works with back-pressure, but can
> alter with @TriggerWhenAnyDestinationAvailable annotation. DistributeLoad is 
> the only processor annotated with this)
> We could use this mechanism to keep the source processor waiting to be 
> scheduled, by following flow:
> {code}
> GetSQS
>   -- success --> FetchS3Object --> Parse --> Notify
>   -- success --> Wait
> {code}
> To make it work as expected, we need to improve Wait so that user can choose 
> how waiting FlowFile is handled, from either:
> "Route to 'wait' relationship" or "Keep in the Upstream connection".
> Currently it has only option to route to 'wait'.
> Use "Keep in the Upstream connection" Wait mode with the flow above,
> the incoming flow file in GetSQS -> Wait connection stays there until actual 
> data processing finishes and Notify sends a notification signal.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3452) Add Wait processor Wait Mode property

[Joseph Gresock (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859437#comment-15859437
 ] 

Joseph Gresock commented on NIFI-3452:
--

Wow, this is a really clever application of Wait/Notify.  Nice addition to 
Wait, as well.

> Add Wait processor Wait Mode property
> -
>
> Key: NIFI-3452
> URL: https://issues.apache.org/jira/browse/NIFI-3452
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Attachments: wait-for-a-par-of-flow-to-finish.png
>
>
> NiFi back pressure is handled per relationship and as long as a relationship 
> has room to receive more flow files, source processor is scheduled to run.
> However, this behavior is not ideal in some cases. For example, when there is 
> very computationally expensive task and user wants to limit the number of 
> FlowFiles can be processed at a given time, it's not always possible to limit 
> the rate by existing RateControl nor back-pressure mechanism.
> As a more practical example, in the following flow, it's expected the GetSQS 
> is triggered only when the previous FlowFile has been processed completely.
> Node 1 is parsing a flow file (indicated by the X in the connection between 
> FetchS3Object and Parse). Both connections have a back-pressure threshold of 
> 1, but because the object is already fetched, the first connection is empty 
> and can thus be filled. This means that, if a new item becomes available in 
> the queue, both of the following cases can happen with equal probability:
> {code}
> Case 1:
> --  -  -
> Node 1: | GetSQS | -X-> | FetchS3Object | -X-> | Parse |
> --  -  -
> --  -  -
> Node 2: | GetSQS | ---> | FetchS3Object | ---> | Parse |
> --  -  -
> Case 2:
> --  -  -
> Node 1: | GetSQS | ---> | FetchS3Object | -X-> | Parse |
> --  -  -
> --  -  -
> Node 2: | GetSQS | -X-> | FetchS3Object | ---> | Parse |
> --  -  -
> {code}
> To achieve that, we could improve Wait processor as follows.
> NiFi scheduler checks downstream relationship availability, when it's full, 
> the processor won't be scheduled to run. In case a source processor has 
> multiple outgoing relationships, and if ANY of those is full, the processor 
> won't be scheduled.
> (This is how processor scheduling works with back-pressure, but can
> alter with @TriggerWhenAnyDestinationAvailable annotation. DistributeLoad is 
> the only processor annotated with this)
> We could use this mechanism to keep the source processor waiting to be 
> scheduled, by following flow:
> {code}
> GetSQS
>   -- success --> FetchS3Object --> Parse --> Notify
>   -- success --> Wait
> {code}
> To make it work as expected, we need to improve Wait so that user can choose 
> how waiting FlowFile is handled, from either:
> "Route to 'wait' relationship" or "Keep in the Upstream connection".
> Currently it has only option to route to 'wait'.
> Use "Keep in the Upstream connection" Wait mode with the flow above,
> the incoming flow file in GetSQS -> Wait connection stays there until actual 
> data processing finishes and Notify sends a notification signal.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3452) Add Wait processor Wait Mode property

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859432#comment-15859432
 ] 

ASF GitHub Bot commented on NIFI-3452:
--

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/1490

NIFI-3452: Add Wait processor Wait Mode property

Ensure back-pressure is active until downstream processing completes.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-3452

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1490


commit 8767b8bf98c83e0e84e68fb53c51255fec4ab084
Author: Koji Kawamura 
Date:   2017-02-09T12:30:43Z

NIFI-3452: Add Wait processor Wait Mode property

Ensure back-pressure is active until downstream processing completes.




> Add Wait processor Wait Mode property
> -
>
> Key: NIFI-3452
> URL: https://issues.apache.org/jira/browse/NIFI-3452
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Attachments: wait-for-a-par-of-flow-to-finish.png
>
>
> NiFi back pressure is handled per relationship and as long as a relationship 
> has room to receive more flow files, source processor is scheduled to run.
> However, this behavior is not ideal in some cases. For example, when there is 
> very computationally expensive task and user wants to limit the number of 
> FlowFiles can be processed at a given time, it's not always possible to limit 
> the rate by existing RateControl nor back-pressure mechanism.
> As a more practical example, in the following flow, it's expected the GetSQS 
> is triggered only when the previous FlowFile has been processed completely.
> Node 1 is parsing a flow file (indicated by the X in the connection between 
> FetchS3Object and Parse). Both connections have a back-pressure threshold of 
> 1, but because the object is already fetched, the first connection is empty 
> and can thus be filled. This means that, if a new item becomes available in 
> the queue, both of the following cases can happen with equal probability:
> {code}
> Case 1:
> --  -  -
> Node 1: | GetSQS | -X-> | FetchS3Object | -X-> | Parse |
> --  -  -
> --  -  -
> Node 2: | GetSQS | ---> | FetchS3Object | ---> | Parse |
> --  -  -
> Case 2:
> --  -  -
> Node 1: | GetSQS | ---> | FetchS3Object | -X-> | Parse |
> --  -  -
> --  -   

[jira] [Updated] (NIFI-3452) Add Wait processor Wait Mode property

[Koji Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-3452:

Status: Patch Available  (was: In Progress)

> Add Wait processor Wait Mode property
> -
>
> Key: NIFI-3452
> URL: https://issues.apache.org/jira/browse/NIFI-3452
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Attachments: wait-for-a-par-of-flow-to-finish.png
>
>
> NiFi back pressure is handled per relationship and as long as a relationship 
> has room to receive more flow files, source processor is scheduled to run.
> However, this behavior is not ideal in some cases. For example, when there is 
> very computationally expensive task and user wants to limit the number of 
> FlowFiles can be processed at a given time, it's not always possible to limit 
> the rate by existing RateControl nor back-pressure mechanism.
> As a more practical example, in the following flow, it's expected the GetSQS 
> is triggered only when the previous FlowFile has been processed completely.
> Node 1 is parsing a flow file (indicated by the X in the connection between 
> FetchS3Object and Parse). Both connections have a back-pressure threshold of 
> 1, but because the object is already fetched, the first connection is empty 
> and can thus be filled. This means that, if a new item becomes available in 
> the queue, both of the following cases can happen with equal probability:
> {code}
> Case 1:
> --  -  -
> Node 1: | GetSQS | -X-> | FetchS3Object | -X-> | Parse |
> --  -  -
> --  -  -
> Node 2: | GetSQS | ---> | FetchS3Object | ---> | Parse |
> --  -  -
> Case 2:
> --  -  -
> Node 1: | GetSQS | ---> | FetchS3Object | -X-> | Parse |
> --  -  -
> --  -  -
> Node 2: | GetSQS | -X-> | FetchS3Object | ---> | Parse |
> --  -  -
> {code}
> To achieve that, we could improve Wait processor as follows.
> NiFi scheduler checks downstream relationship availability, when it's full, 
> the processor won't be scheduled to run. In case a source processor has 
> multiple outgoing relationships, and if ANY of those is full, the processor 
> won't be scheduled.
> (This is how processor scheduling works with back-pressure, but can
> alter with @TriggerWhenAnyDestinationAvailable annotation. DistributeLoad is 
> the only processor annotated with this)
> We could use this mechanism to keep the source processor waiting to be 
> scheduled, by following flow:
> {code}
> GetSQS
>   -- success --> FetchS3Object --> Parse --> Notify
>   -- success --> Wait
> {code}
> To make it work as expected, we need to improve Wait so that user can choose 
> how waiting FlowFile is handled, from either:
> "Route to 'wait' relationship" or "Keep in the Upstream connection".
> Currently it has only option to route to 'wait'.
> Use "Keep in the Upstream connection" Wait mode with the flow above,
> the incoming flow file in GetSQS -> Wait connection stays there until actual 
> data processing finishes and Notify sends a notification signal.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1490: NIFI-3452: Add Wait processor Wait Mode property

[ijokarumawak]

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/1490

NIFI-3452: Add Wait processor Wait Mode property

Ensure back-pressure is active until downstream processing completes.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-3452

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1490


commit 8767b8bf98c83e0e84e68fb53c51255fec4ab084
Author: Koji Kawamura 
Date:   2017-02-09T12:30:43Z

NIFI-3452: Add Wait processor Wait Mode property

Ensure back-pressure is active until downstream processing completes.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3452) Add Wait processor Wait Mode property

[Koji Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-3452:

Attachment: wait-for-a-par-of-flow-to-finish.png

> Add Wait processor Wait Mode property
> -
>
> Key: NIFI-3452
> URL: https://issues.apache.org/jira/browse/NIFI-3452
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Attachments: wait-for-a-par-of-flow-to-finish.png
>
>
> NiFi back pressure is handled per relationship and as long as a relationship 
> has room to receive more flow files, source processor is scheduled to run.
> However, this behavior is not ideal in some cases. For example, when there is 
> very computationally expensive task and user wants to limit the number of 
> FlowFiles can be processed at a given time, it's not always possible to limit 
> the rate by existing RateControl nor back-pressure mechanism.
> As a more practical example, in the following flow, it's expected the GetSQS 
> is triggered only when the previous FlowFile has been processed completely.
> Node 1 is parsing a flow file (indicated by the X in the connection between 
> FetchS3Object and Parse). Both connections have a back-pressure threshold of 
> 1, but because the object is already fetched, the first connection is empty 
> and can thus be filled. This means that, if a new item becomes available in 
> the queue, both of the following cases can happen with equal probability:
> {code}
> Case 1:
> --  -  -
> Node 1: | GetSQS | -X-> | FetchS3Object | -X-> | Parse |
> --  -  -
> --  -  -
> Node 2: | GetSQS | ---> | FetchS3Object | ---> | Parse |
> --  -  -
> Case 2:
> --  -  -
> Node 1: | GetSQS | ---> | FetchS3Object | -X-> | Parse |
> --  -  -
> --  -  -
> Node 2: | GetSQS | -X-> | FetchS3Object | ---> | Parse |
> --  -  -
> {code}
> To achieve that, we could improve Wait processor as follows.
> NiFi scheduler checks downstream relationship availability, when it's full, 
> the processor won't be scheduled to run. In case a source processor has 
> multiple outgoing relationships, and if ANY of those is full, the processor 
> won't be scheduled.
> (This is how processor scheduling works with back-pressure, but can
> alter with @TriggerWhenAnyDestinationAvailable annotation. DistributeLoad is 
> the only processor annotated with this)
> We could use this mechanism to keep the source processor waiting to be 
> scheduled, by following flow:
> {code}
> GetSQS
>   -- success --> FetchS3Object --> Parse --> Notify
>   -- success --> Wait
> {code}
> To make it work as expected, we need to improve Wait so that user can choose 
> how waiting FlowFile is handled, from either:
> "Route to 'wait' relationship" or "Keep in the Upstream connection".
> Currently it has only option to route to 'wait'.
> Use "Keep in the Upstream connection" Wait mode with the flow above,
> the incoming flow file in GetSQS -> Wait connection stays there until actual 
> data processing finishes and Notify sends a notification signal.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)