date:20161012


[ 
https://issues.apache.org/jira/browse/NIFI-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569492#comment-15569492
 ] 

ASF GitHub Bot commented on NIFI-2799:
--

Github user ktseytlin commented on the issue:

https://github.com/apache/nifi/pull/1112
  
@jvwing I tested with the updated logic on my end and it worked, so I 
pushed that out in the most recent commit. Does all look good to you now? Any 
other improvements we can make?


> AWS Credentials for Assume Role Need Proxy
> --
>
> Key: NIFI-2799
> URL: https://issues.apache.org/jira/browse/NIFI-2799
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Keren Tseytlin
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.1.0
>
>
> As a user of Nifi, when I want to enable cross account fetching of S3 objects 
> I need the proxy variables to be set in order to generate temporary AWS 
> tokens for STS:AssumeRole.
> Within some enterprise environments, it is necessary to set the proxy 
> variables prior to running AssumeRole methods. Without this being set, the 
> machine in VPC A times out on generating temporary keys and is unable to 
> assume a role as a machine in VPC B. 
> This ticket arose from this conversation: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-Cross-Account-Download-With-A-Profile-Flag-td13232.html#a13252
> Goal: There are files stored in an S3 bucket in VPC B. My Nifi machines are 
> in VPC A. I want Nifi to be able to get those files from VPC B. VPC A and VPC 
> B need to be communicating in the FetchS3Object component.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1119: NIFI-2801 Edited Kafka processor documentation to e...

2016-10-12 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1119


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1119: NIFI-2801 Edited Kafka processor documentation to explicit...

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/1119
  
Reviewed, ran `contrib-check`, and added each processor to an instance of 
NiFi. Verified the documentation was present. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2801) Add information about relevant version to *Kafka processor documentation


[ 
https://issues.apache.org/jira/browse/NIFI-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569646#comment-15569646
 ] 

ASF GitHub Bot commented on NIFI-2801:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1119#discussion_r83079539
  
--- Diff: nifi-docs/src/main/asciidoc/getting-started.adoc ---
@@ -386,7 +385,7 @@ categorizing them by their functions.
 - *PutSQL*: Executes the contents of a FlowFile as a SQL DDL Statement 
(INSERT, UPDATE, or DELETE). The contents of the FlowFile must be a valid
   SQL statement. Attributes can be used as parameters so that the contents 
of the FlowFile can be parameterized SQL statements in order to avoid
SQL injection attacks.
-- *PutKafka*: Sends the contents of a FlowFile to Kafka as a message. The 
FlowFile can be sent as a single message or a delimiter, such as a
+- *PutKafka*: Sends the contents of a FlowFile as a message to Apache 
Kafka, specifically for 0.8.x versions. The FlowFile can be sent as a single 
message or a delimiter, such as a
--- End diff --

I confirmed that these processors only work for `0.8.x`. 


> Add information about relevant version to *Kafka processor documentation
> 
>
> Key: NIFI-2801
> URL: https://issues.apache.org/jira/browse/NIFI-2801
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andy LoPresto
>Assignee: Andrew Lim
>Priority: Minor
>  Labels: Beginner
Documentation
Kafka
> Fix For: 1.1.0, 0.8.0
>
>
> A frequent obstacle for new users is the various processors for communicating 
> with Kafka. Due to incompatibilities between different versions of Kafka, 
> there are currently 6 processors (3 to push data and 3 to pull data), and 
> each "pair" targets a specific version. We should add text to the 
> documentation/description of each to clarify explicitly which version of 
> Kafka the processor targets, and what its "complementary" processor is named. 
> * <=0.8 -- PutKafka/GetKafka
> * 0.9 -- PublishKafka/ConsumeKafka
> * 0.10 -- PublishKafka_0_10/ConsumeKafka_0_10



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2801) Add information about relevant version to *Kafka processor documentation

2016-10-12 Thread Andy LoPresto (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569651#comment-15569651
 ] 

Andy LoPresto commented on NIFI-2801:
-

Merged 1119 into master; will do 1124 (0.x) later today. 

> Add information about relevant version to *Kafka processor documentation
> 
>
> Key: NIFI-2801
> URL: https://issues.apache.org/jira/browse/NIFI-2801
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andy LoPresto
>Assignee: Andrew Lim
>Priority: Minor
>  Labels: Beginner
Documentation
Kafka
> Fix For: 1.1.0, 0.8.0
>
>
> A frequent obstacle for new users is the various processors for communicating 
> with Kafka. Due to incompatibilities between different versions of Kafka, 
> there are currently 6 processors (3 to push data and 3 to pull data), and 
> each "pair" targets a specific version. We should add text to the 
> documentation/description of each to clarify explicitly which version of 
> Kafka the processor targets, and what its "complementary" processor is named. 
> * <=0.8 -- PutKafka/GetKafka
> * 0.9 -- PublishKafka/ConsumeKafka
> * 0.10 -- PublishKafka_0_10/ConsumeKafka_0_10



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2801) Add information about relevant version to *Kafka processor documentation


[ 
https://issues.apache.org/jira/browse/NIFI-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569644#comment-15569644
 ] 

ASF GitHub Bot commented on NIFI-2801:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1119


> Add information about relevant version to *Kafka processor documentation
> 
>
> Key: NIFI-2801
> URL: https://issues.apache.org/jira/browse/NIFI-2801
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andy LoPresto
>Assignee: Andrew Lim
>Priority: Minor
>  Labels: Beginner
Documentation
Kafka
> Fix For: 1.1.0, 0.8.0
>
>
> A frequent obstacle for new users is the various processors for communicating 
> with Kafka. Due to incompatibilities between different versions of Kafka, 
> there are currently 6 processors (3 to push data and 3 to pull data), and 
> each "pair" targets a specific version. We should add text to the 
> documentation/description of each to clarify explicitly which version of 
> Kafka the processor targets, and what its "complementary" processor is named. 
> * <=0.8 -- PutKafka/GetKafka
> * 0.9 -- PublishKafka/ConsumeKafka
> * 0.10 -- PublishKafka_0_10/ConsumeKafka_0_10



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1119: NIFI-2801 Edited Kafka processor documentation to e...

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1119#discussion_r83079553
  
--- Diff: nifi-docs/src/main/asciidoc/getting-started.adoc ---
@@ -371,8 +371,7 @@ categorizing them by their functions.
the content fetched from HDFS.
 - *FetchS3Object*: Fetches the contents of an object from the Amazon Web 
Services (AWS) Simple Storage Service (S3). The outbound FlowFile contains the 
contents
   received from S3.
-- *GetKafka*: Consumes messages from Apache Kafka. The messages can be 
emitted as a FlowFile per message or can be batched together using a 
user-specified
-  delimiter.
+- *GetKafka*: Fetches messages from Apache Kafka, specifically for 0.8.x 
versions. The messages can be emitted as a FlowFile per message or can be 
batched together using a user-specified delimiter.
--- End diff --

I confirmed that these processors only work for `0.8.x`. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1119: NIFI-2801 Edited Kafka processor documentation to e...

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1119#discussion_r83079539
  
--- Diff: nifi-docs/src/main/asciidoc/getting-started.adoc ---
@@ -386,7 +385,7 @@ categorizing them by their functions.
 - *PutSQL*: Executes the contents of a FlowFile as a SQL DDL Statement 
(INSERT, UPDATE, or DELETE). The contents of the FlowFile must be a valid
   SQL statement. Attributes can be used as parameters so that the contents 
of the FlowFile can be parameterized SQL statements in order to avoid
SQL injection attacks.
-- *PutKafka*: Sends the contents of a FlowFile to Kafka as a message. The 
FlowFile can be sent as a single message or a delimiter, such as a
+- *PutKafka*: Sends the contents of a FlowFile as a message to Apache 
Kafka, specifically for 0.8.x versions. The FlowFile can be sent as a single 
message or a delimiter, such as a
--- End diff --

I confirmed that these processors only work for `0.8.x`. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2801) Add information about relevant version to *Kafka processor documentation


[ 
https://issues.apache.org/jira/browse/NIFI-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569647#comment-15569647
 ] 

ASF GitHub Bot commented on NIFI-2801:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1119#discussion_r83079553
  
--- Diff: nifi-docs/src/main/asciidoc/getting-started.adoc ---
@@ -371,8 +371,7 @@ categorizing them by their functions.
the content fetched from HDFS.
 - *FetchS3Object*: Fetches the contents of an object from the Amazon Web 
Services (AWS) Simple Storage Service (S3). The outbound FlowFile contains the 
contents
   received from S3.
-- *GetKafka*: Consumes messages from Apache Kafka. The messages can be 
emitted as a FlowFile per message or can be batched together using a 
user-specified
-  delimiter.
+- *GetKafka*: Fetches messages from Apache Kafka, specifically for 0.8.x 
versions. The messages can be emitted as a FlowFile per message or can be 
batched together using a user-specified delimiter.
--- End diff --

I confirmed that these processors only work for `0.8.x`. 


> Add information about relevant version to *Kafka processor documentation
> 
>
> Key: NIFI-2801
> URL: https://issues.apache.org/jira/browse/NIFI-2801
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andy LoPresto
>Assignee: Andrew Lim
>Priority: Minor
>  Labels: Beginner
Documentation
Kafka
> Fix For: 1.1.0, 0.8.0
>
>
> A frequent obstacle for new users is the various processors for communicating 
> with Kafka. Due to incompatibilities between different versions of Kafka, 
> there are currently 6 processors (3 to push data and 3 to pull data), and 
> each "pair" targets a specific version. We should add text to the 
> documentation/description of each to clarify explicitly which version of 
> Kafka the processor targets, and what its "complementary" processor is named. 
> * <=0.8 -- PutKafka/GetKafka
> * 0.9 -- PublishKafka/ConsumeKafka
> * 0.10 -- PublishKafka_0_10/ConsumeKafka_0_10



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1125: NIFI-2872 Create PutCloudwatchMetric Processor

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1125#discussion_r83091812
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/cloudwatch/PutCloudWatchMetric.java
 ---
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.cloudwatch;
+
+import com.amazonaws.ClientConfiguration;
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.services.cloudwatch.AmazonCloudWatchClient;
+import com.amazonaws.services.cloudwatch.model.MetricDatum;
+import com.amazonaws.services.cloudwatch.model.PutMetricDataRequest;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"amazon", "aws", "cloudwatch", "metrics", "put", "publish"})
+@CapabilityDescription("Publishes metrics to Amazon CloudWatch")
+public class PutCloudWatchMetric extends 
AbstractAWSCredentialsProviderProcessor {
+
+public static final Set relationships = 
Collections.unmodifiableSet(
+new HashSet<>(Arrays.asList(REL_SUCCESS, REL_FAILURE)));
+
+private static final Validator DOUBLE_VALIDATOR = new Validator() {
+@Override
+public ValidationResult validate(String subject, String input, 
ValidationContext context) {
+if (context.isExpressionLanguageSupported(subject) && 
context.isExpressionLanguagePresent(input)) {
+return (new 
ValidationResult.Builder()).subject(subject).input(input).explanation("Expression
 Language Present").valid(true).build();
+} else {
+String reason = null;
+
+try {
+Double.parseDouble(input);
+} catch (NumberFormatException var6) {
--- End diff --

Any significance to `var6`?  How about `nfe` or just `e`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2872) Create PutCloudwatchMetric Processor


[ 
https://issues.apache.org/jira/browse/NIFI-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569803#comment-15569803
 ] 

ASF GitHub Bot commented on NIFI-2872:
--

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1125#discussion_r83092101
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/cloudwatch/PutCloudWatchMetric.java
 ---
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.cloudwatch;
+
+import com.amazonaws.ClientConfiguration;
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.services.cloudwatch.AmazonCloudWatchClient;
+import com.amazonaws.services.cloudwatch.model.MetricDatum;
+import com.amazonaws.services.cloudwatch.model.PutMetricDataRequest;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"amazon", "aws", "cloudwatch", "metrics", "put", "publish"})
+@CapabilityDescription("Publishes metrics to Amazon CloudWatch")
+public class PutCloudWatchMetric extends 
AbstractAWSCredentialsProviderProcessor {
+
+public static final Set relationships = 
Collections.unmodifiableSet(
+new HashSet<>(Arrays.asList(REL_SUCCESS, REL_FAILURE)));
+
+private static final Validator DOUBLE_VALIDATOR = new Validator() {
+@Override
+public ValidationResult validate(String subject, String input, 
ValidationContext context) {
+if (context.isExpressionLanguageSupported(subject) && 
context.isExpressionLanguagePresent(input)) {
+return (new 
ValidationResult.Builder()).subject(subject).input(input).explanation("Expression
 Language Present").valid(true).build();
+} else {
+String reason = null;
+
+try {
+Double.parseDouble(input);
+} catch (NumberFormatException var6) {
+reason = "not a valid Double";
+}
+
+return (new 
ValidationResult.Builder()).subject(subject).input(input).explanation(reason).valid(reason
 == null).build();
+}
+}
+};
+
+public static final PropertyDescriptor NAMESPACE = new 
PropertyDescriptor.Builder()
+.name("Namespace")
+.displayName("Namespace")
+.description("The namespace for the metric data for 
CloudWatch")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor METRIC_NAME = new

[GitHub] nifi pull request #1125: NIFI-2872 Create PutCloudwatchMetric Processor

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1125#discussion_r83092101
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/cloudwatch/PutCloudWatchMetric.java
 ---
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.cloudwatch;
+
+import com.amazonaws.ClientConfiguration;
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.services.cloudwatch.AmazonCloudWatchClient;
+import com.amazonaws.services.cloudwatch.model.MetricDatum;
+import com.amazonaws.services.cloudwatch.model.PutMetricDataRequest;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"amazon", "aws", "cloudwatch", "metrics", "put", "publish"})
+@CapabilityDescription("Publishes metrics to Amazon CloudWatch")
+public class PutCloudWatchMetric extends 
AbstractAWSCredentialsProviderProcessor {
+
+public static final Set relationships = 
Collections.unmodifiableSet(
+new HashSet<>(Arrays.asList(REL_SUCCESS, REL_FAILURE)));
+
+private static final Validator DOUBLE_VALIDATOR = new Validator() {
+@Override
+public ValidationResult validate(String subject, String input, 
ValidationContext context) {
+if (context.isExpressionLanguageSupported(subject) && 
context.isExpressionLanguagePresent(input)) {
+return (new 
ValidationResult.Builder()).subject(subject).input(input).explanation("Expression
 Language Present").valid(true).build();
+} else {
+String reason = null;
+
+try {
+Double.parseDouble(input);
+} catch (NumberFormatException var6) {
+reason = "not a valid Double";
+}
+
+return (new 
ValidationResult.Builder()).subject(subject).input(input).explanation(reason).valid(reason
 == null).build();
+}
+}
+};
+
+public static final PropertyDescriptor NAMESPACE = new 
PropertyDescriptor.Builder()
+.name("Namespace")
+.displayName("Namespace")
+.description("The namespace for the metric data for 
CloudWatch")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor METRIC_NAME = new 
PropertyDescriptor.Builder()
+.name("MetricName")
+.displayName("MetricName")
+.description("The name of the metric")
+.expressionLanguageSupported(true)
+

[jira] [Commented] (NIFI-2872) Create PutCloudwatchMetric Processor


[ 
https://issues.apache.org/jira/browse/NIFI-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569834#comment-15569834
 ] 

ASF GitHub Bot commented on NIFI-2872:
--

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/1125
  
@vegaed Thanks for the updated code.  The situation with 
AbstractAWSProcessor is a bit odd.  It's deprecated in that you should not 
derive directly from it, but it still has live code we're definitely using.

* **Number Parsing Exception Handling** - I tested with a non-double metric 
value.  A java.lang.NumberFormatException is thrown parsing the number at 
PutCloudWatchMetric.java:174.  The error message is clear enough, but the 
exception is not caught inside onTrigger(), so the entire session is rolled 
back and the flowfile remains in the incoming queue.  The same thing applies to 
parsing Timestamp.  Shouldn't we catch the exception and route the flowfile to 
failure in a case like this?
* **Integration Tests** - Integration tests failed for me, complaining 
about no flowfiles routed to REL_SUCCESS.  I believe the fix might be as simple 
as adding an empty flowfile before running:

runner.enqueue(new byte[] {});
runner.run();

* **Unit Tests** - How about some unit tests that do not call AWS?  I agree 
there is no substitute for integration tests that actually call AWS, but I 
believe there is also no substitute for unit tests that can easily be run by 
anyone and everyone.  This can be a bit tricky for cloud services, but it seems 
like CloudWatch has an API with nice request/response patterns that can be 
mocked.  So I tried my hand at a [simple mock setup with sample tests for 
PutCloudWatchMetric](https://github.com/jvwing/nifi/commit/b6ae1accfecb464fe16a456cdef04669a65672fe).
  You're welcome to not use this, but I recommend we try something similar that 
gets coverage of onTrigger from unit tests.


> Create PutCloudwatchMetric Processor
> 
>
> Key: NIFI-2872
> URL: https://issues.apache.org/jira/browse/NIFI-2872
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core Framework
>Affects Versions: 1.1.0
>Reporter: Edgardo Vega
>Priority: Minor
>
> Create a processor for put metrics into aws cloudwatch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1125: NIFI-2872 Create PutCloudwatchMetric Processor

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/1125

@vegaed Thanks for the updated code. The situation with
AbstractAWSProcessor is a bit odd. It's deprecated in that you should not
derive directly from it, but it still has live code we're definitely using.

* **Number Parsing Exception Handling** - I tested with a non-double metric
value. A java.lang.NumberFormatException is thrown parsing the number at
PutCloudWatchMetric.java:174. The error message is clear enough, but the
exception is not caught inside onTrigger(), so the entire session is rolled
back and the flowfile remains in the incoming queue. The same thing applies to
parsing Timestamp. Shouldn't we catch the exception and route the flowfile to
failure in a case like this?
* **Integration Tests** - Integration tests failed for me, complaining
about no flowfiles routed to REL_SUCCESS. I believe the fix might be as simple
as adding an empty flowfile before running:

runner.enqueue(new byte[] {});
runner.run();

* **Unit Tests** - How about some unit tests that do not call AWS? I agree
there is no substitute for integration tests that actually call AWS, but I
believe there is also no substitute for unit tests that can easily be run by
anyone and everyone. This can be a bit tricky for cloud services, but it seems
like CloudWatch has an API with nice request/response patterns that can be
mocked. So I tried my hand at a [simple mock setup with sample tests for
PutCloudWatchMetric](https://github.com/jvwing/nifi/commit/b6ae1accfecb464fe16a456cdef04669a65672fe).
You're welcome to not use this, but I recommend we try something similar that
gets coverage of onTrigger from unit tests.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2872) Create PutCloudwatchMetric Processor


[ 
https://issues.apache.org/jira/browse/NIFI-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569802#comment-15569802
 ] 

ASF GitHub Bot commented on NIFI-2872:
--

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1125#discussion_r83091812
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/cloudwatch/PutCloudWatchMetric.java
 ---
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.cloudwatch;
+
+import com.amazonaws.ClientConfiguration;
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.services.cloudwatch.AmazonCloudWatchClient;
+import com.amazonaws.services.cloudwatch.model.MetricDatum;
+import com.amazonaws.services.cloudwatch.model.PutMetricDataRequest;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"amazon", "aws", "cloudwatch", "metrics", "put", "publish"})
+@CapabilityDescription("Publishes metrics to Amazon CloudWatch")
+public class PutCloudWatchMetric extends 
AbstractAWSCredentialsProviderProcessor {
+
+public static final Set relationships = 
Collections.unmodifiableSet(
+new HashSet<>(Arrays.asList(REL_SUCCESS, REL_FAILURE)));
+
+private static final Validator DOUBLE_VALIDATOR = new Validator() {
+@Override
+public ValidationResult validate(String subject, String input, 
ValidationContext context) {
+if (context.isExpressionLanguageSupported(subject) && 
context.isExpressionLanguagePresent(input)) {
+return (new 
ValidationResult.Builder()).subject(subject).input(input).explanation("Expression
 Language Present").valid(true).build();
+} else {
+String reason = null;
+
+try {
+Double.parseDouble(input);
+} catch (NumberFormatException var6) {
--- End diff --

Any significance to `var6`?  How about `nfe` or just `e`?


> Create PutCloudwatchMetric Processor
> 
>
> Key: NIFI-2872
> URL: https://issues.apache.org/jira/browse/NIFI-2872
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core Framework
>Affects Versions: 1.1.0
>Reporter: Edgardo Vega
>Priority: Minor
>
> Create a processor for put metrics into aws cloudwatch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi-minifi pull request #42: MINIFI-120 Fixing 'class not found' issues whe...

2016-10-12 Thread JPercivall

GitHub user JPercivall opened a pull request:

https://github.com/apache/nifi-minifi/pull/42

MINIFI-120 Fixing 'class not found' issues when using the Rest Changeâ¦

â¦ Notifier. Also added all the jetty and javax modules that NiFi lists as 
'provided' in its pom to the minifi-assembly pom in order to ensure NAR 
reusability.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JPercivall/nifi-minifi MINIFI-120

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi/pull/42.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #42


commit 499c65510e8530ddeb0a7f9baa6a481dc491f9cb
Author: Joseph Percivall 
Date:   2016-10-12T21:08:49Z

MINIFI-120 Fixing 'class not found' issues when using the Rest Change 
Notifier. Also added all the jetty and javax modules that NiFi lists as 
'provided' in its pom to the minifi-assembly pom in order to ensure NAR 
reusability.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2199) NiFi cannot be restarted through SSH (nifi.sh hangs)

2016-10-12 Thread Michal Klempa (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567982#comment-15567982
 ] 

Michal Klempa commented on NIFI-2199:
-

[~trixpan] yes, i have the fixes for all versions since 0.5.0, but it is 
possible to submit a 'hotfix' to a released version? If it is so, tell me the 
right branches to which I should propose pull requests then and I will prepare 
pull requests.

> NiFi cannot be restarted through SSH (nifi.sh hangs)
> 
>
> Key: NIFI-2199
> URL: https://issues.apache.org/jira/browse/NIFI-2199
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0, 0.5.1, 0.7.0, 0.6.1
>Reporter: Michal Klempa
>Priority: Minor
>  Labels: initscript
>
> When NiFi is to restarted using SSH from remote host, nifi.sh causes SSH to 
> hang on open io streams.
> Steps to reproduce:
> 1. HostA - NiFi instalation
> 2. HostB - client machine
> 3. log to HostB
> 4. issue a command
> {code}
> ssh root@HostA /etc/init.d/nifi restart
> {code}
> Expected behavior: NiFi is restarted, SSH command exits with 0
> Actual behavior: NiFi is restarted, SSH command hang on
> This seems to be somehow related to SSH handling IO streams of process. Seems 
> like detached NiFi java process does not close stdin/out/err (i do not know 
> precisely) leading to sshd waiting for some input/output to come to client.
> I was able to do a quick fix in nifi.sh as follows:
> 1. on existing nifi installation, in file /etc/init.d/nifi find the line
> {code}
> (cd "${NIFI_HOME}" && ${sudo_cmd_prefix} "${JAVA}" -cp 
> "${BOOTSTRAP_CLASSPATH}" -Xms12m -Xmx24m 
> -Dorg.apache.nifi.bootstrap.config.file="${BOOTSTRAP_CONF}" 
> org.apache.nifi.bootstrap.RunNiFi $@ &)
> {code}
> its line number 214 (true branch of if [ "$1"] = "start"] condition).
> 2. replace it by line:
> {code}
> (cd "${NIFI_HOME}" && ${sudo_cmd_prefix} "${JAVA}" -cp 
> "${BOOTSTRAP_CLASSPATH}" -Xms12m -Xmx24m 
> -Dorg.apache.nifi.bootstrap.config.file="${BOOTSTRAP_CONF}" 
> org.apache.nifi.bootstrap.RunNiFi $@ > /dev/null 2>&1 < /dev/null &)
> {code}
> This simply redirects stdin from /dev/null to NiFi java process and redirects 
> stdout/err to /dev/null letting SSH to close the session.
> After applying this fix, I am able to restart NiFi remotely using SSH.
> Use case for this scenario: When Flow Manager develop flow.xml.gz on 
> localhost/testing environments, usually deployment to production environment 
> is handled by jenkins/bamboo CI tools. These tools need to log in, upload new 
> flow.xml.gz and restart NiFi. All remotely using SSH.
> If somebody can advice me where in the source code is this line of code 
> located, I can prepare a pull request for this quick fix.
> But. Maybe we may want to go further and investigate what to do with IO 
> streams inside Java...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2199) NiFi cannot be restarted through SSH (nifi.sh hangs)

2016-10-12 Thread Michal Klempa (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michal Klempa updated NIFI-2199:

Affects Version/s: 1.0.0

> NiFi cannot be restarted through SSH (nifi.sh hangs)
> 
>
> Key: NIFI-2199
> URL: https://issues.apache.org/jira/browse/NIFI-2199
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0, 0.5.1, 0.7.0, 0.6.1
>Reporter: Michal Klempa
>Priority: Minor
>  Labels: initscript
>
> When NiFi is to restarted using SSH from remote host, nifi.sh causes SSH to 
> hang on open io streams.
> Steps to reproduce:
> 1. HostA - NiFi instalation
> 2. HostB - client machine
> 3. log to HostB
> 4. issue a command
> {code}
> ssh root@HostA /etc/init.d/nifi restart
> {code}
> Expected behavior: NiFi is restarted, SSH command exits with 0
> Actual behavior: NiFi is restarted, SSH command hang on
> This seems to be somehow related to SSH handling IO streams of process. Seems 
> like detached NiFi java process does not close stdin/out/err (i do not know 
> precisely) leading to sshd waiting for some input/output to come to client.
> I was able to do a quick fix in nifi.sh as follows:
> 1. on existing nifi installation, in file /etc/init.d/nifi find the line
> {code}
> (cd "${NIFI_HOME}" && ${sudo_cmd_prefix} "${JAVA}" -cp 
> "${BOOTSTRAP_CLASSPATH}" -Xms12m -Xmx24m 
> -Dorg.apache.nifi.bootstrap.config.file="${BOOTSTRAP_CONF}" 
> org.apache.nifi.bootstrap.RunNiFi $@ &)
> {code}
> its line number 214 (true branch of if [ "$1"] = "start"] condition).
> 2. replace it by line:
> {code}
> (cd "${NIFI_HOME}" && ${sudo_cmd_prefix} "${JAVA}" -cp 
> "${BOOTSTRAP_CLASSPATH}" -Xms12m -Xmx24m 
> -Dorg.apache.nifi.bootstrap.config.file="${BOOTSTRAP_CONF}" 
> org.apache.nifi.bootstrap.RunNiFi $@ > /dev/null 2>&1 < /dev/null &)
> {code}
> This simply redirects stdin from /dev/null to NiFi java process and redirects 
> stdout/err to /dev/null letting SSH to close the session.
> After applying this fix, I am able to restart NiFi remotely using SSH.
> Use case for this scenario: When Flow Manager develop flow.xml.gz on 
> localhost/testing environments, usually deployment to production environment 
> is handled by jenkins/bamboo CI tools. These tools need to log in, upload new 
> flow.xml.gz and restart NiFi. All remotely using SSH.
> If somebody can advice me where in the source code is this line of code 
> located, I can prepare a pull request for this quick fix.
> But. Maybe we may want to go further and investigate what to do with IO 
> streams inside Java...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #675: LSB Adherence

2016-10-12 Thread michalklempa

Github user michalklempa commented on a diff in the pull request:

https://github.com/apache/nifi/pull/675#discussion_r82953278
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
 ---
@@ -265,20 +265,54 @@ run() {
 BOOTSTRAP_DIR_PARAMS="${BOOTSTRAP_LOG_PARAMS} ${BOOTSTRAP_PID_PARAMS} 
${BOOTSTRAP_CONF_PARAMS}"
 
 RUN_NIFI_CMD="cd "\""${NIFI_HOME}"\"" && ${sudo_cmd_prefix} 
"\""${JAVA}"\"" -cp "\""${BOOTSTRAP_CLASSPATH}"\"" -Xms12m -Xmx24m 
${BOOTSTRAP_DIR_PARAMS}  org.apache.nifi.bootstrap.RunNiFi"
+LSB_EXIT_STATUS=0
+LSB_NIFI_STATUS_TEMPFILE=`mktemp -t 
LSB_NIFI_STATUS_TEMPFILE.XX`
 
 if [ "$1" = "start" ]; then
 (eval $RUN_NIFI_CMD $@ &)
 else
-(eval $RUN_NIFI_CMD $@)
+(eval $RUN_NIFI_CMD $@ "| tee "$LSB_NIFI_STATUS_TEMPFILE)
 fi
 
 # Wait just a bit (3 secs) to wait for the logging to finish and then 
echo a new-line.
 # We do this to avoid having logs spewed on the console after running 
the command and then not giving
 # control back to the user
 sleep 3
 echo
+if [ "$1" = "status" ]; then
+LSB_EXIT_STATUS= getLSBStatus
+fi
+return $LSB_EXIT_STATUS
 }
 
+# Standard Status codes for LSB compliance
+# See: 
http://refspecs.linuxbase.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html
+# If the status action is requested, the init script will return the 
following exit status codes.
+# 0program is running or service is OK
+# 1program is dead and /var/run pid file exists
+# 2program is dead and /var/lock lock file exists
+# 3program is not running
+# 4program or service status is unknown
+# 5-99 reserved for future LSB use
+# 100-149  reserved for distribution use
+# 150-199  reserved for application use
+# 200-254  reserved
+
+getLSBStatus() {
+  LSB_EXIT_STATUS=4
+  if [ `grep -c "not.*running" ${LSB_NIFI_STATUS_TEMPFILE}` -gt 0 ]; then
--- End diff --

Why grep output when we can fix the cause (RunNifi.java) see: 
https://github.com/apache/nifi/pull/1093


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #675: LSB Adherence

2016-10-12 Thread michalklempa

Github user michalklempa commented on the issue:

https://github.com/apache/nifi/pull/675
  
This approach is too complicated. Why clutter with grepping output when we 
can fix the RunNifi.java


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1769) Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS


[ 
https://issues.apache.org/jira/browse/NIFI-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568854#comment-15568854
 ] 

ASF GitHub Bot commented on NIFI-1769:
--

GitHub user miquillo opened a pull request:

https://github.com/apache/nifi/pull/1126

Implemented SSE with KMS. Jira #NIFI-1769

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/miquillo/nifi master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1126.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1126


commit f28b8feb324ed79207e0f03761163c1e8baf64d9
Author: Michiel Moonen 
Date:   2016-10-12T14:13:37Z

implemented SSE with KMS




> Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS
> -
>
> Key: NIFI-1769
> URL: https://issues.apache.org/jira/browse/NIFI-1769
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Michiel Moonen
>Priority: Minor
>  Labels: newbie, patch, security
>
> Currently there is no support for SSE-KMS S3 Signature Version 4 
> Authentication. This is necessary for enhanced security features



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1126: Implemented SSE with KMS. Jira #NIFI-1769

2016-10-12 Thread miquillo

GitHub user miquillo opened a pull request:

https://github.com/apache/nifi/pull/1126

Implemented SSE with KMS. Jira #NIFI-1769

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/miquillo/nifi master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1126.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1126


commit f28b8feb324ed79207e0f03761163c1e8baf64d9
Author: Michiel Moonen 
Date:   2016-10-12T14:13:37Z

implemented SSE with KMS




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (NIFI-2164) ConsumeJMS should allow user to configure trade-off between 'best effort' and 'guaranteed receipt' of data

2016-10-12 Thread Oleg Zhurakousky (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky resolved NIFI-2164.

Resolution: Duplicate

> ConsumeJMS should allow user to configure trade-off between 'best effort' and 
> 'guaranteed receipt' of data
> --
>
> Key: NIFI-2164
> URL: https://issues.apache.org/jira/browse/NIFI-2164
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Oleg Zhurakousky
>
> Currently the ConsumeJMS Processor uses auto-acknowledge acknowledgement. 
> This is beneficial for many use cases but could result in data loss if NiFi 
> is shut down. We should expose a 'Delivery Guarantee' property that allows 
> the user to choose between 'Best Effort', which will provide better 
> performance or 'Guaranteed Receipt', which will guarantee that data has been 
> committed to NiFi's Content & FlowFile Repositories before acknowledging the 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-1769) Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS

2016-10-12 Thread Michiel Moonen (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568864#comment-15568864
 ] 

Michiel Moonen edited comment on NIFI-1769 at 10/12/16 2:25 PM:


Closed down the previous PR because it was too much of a hassle, including too 
many unnecessary commits.
Create a new PR GitHub Pull Request #1126, with a single commit addressing the 
SSE-KMS feature. The S3v4 signature version has already been implemented in a 
previous version.



was (Author: michielmoo...@hotmail.com):
Closed down the previous PR because it was too much of a hassle, including too 
many unnecessary commits.
Create a new PR, with a single commit addressing the SSE-KMS feature. The S3v4 
signature version has already been implemented in a previous version.


> Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS
> -
>
> Key: NIFI-1769
> URL: https://issues.apache.org/jira/browse/NIFI-1769
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Michiel Moonen
>Priority: Minor
>  Labels: newbie, patch, security
>
> Currently there is no support for SSE-KMS S3 Signature Version 4 
> Authentication. This is necessary for enhanced security features



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #362: NIFI-1769: added support for SSE-KMS and signature s...

2016-10-12 Thread miquillo

Github user miquillo closed the pull request at:

https://github.com/apache/nifi/pull/362


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1769) Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS


[ 
https://issues.apache.org/jira/browse/NIFI-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568852#comment-15568852
 ] 

ASF GitHub Bot commented on NIFI-1769:
--

Github user miquillo closed the pull request at:

https://github.com/apache/nifi/pull/362


> Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS
> -
>
> Key: NIFI-1769
> URL: https://issues.apache.org/jira/browse/NIFI-1769
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Michiel Moonen
>Priority: Minor
>  Labels: newbie, patch, security
>
> Currently there is no support for SSE-KMS S3 Signature Version 4 
> Authentication. This is necessary for enhanced security features



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1769) Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS

2016-10-12 Thread Michiel Moonen (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568864#comment-15568864
 ] 

Michiel Moonen commented on NIFI-1769:
--

Closed down the previous PR because it was too much of a hassle, including too 
many unnecessary commits.
Create a new PR, with a single commit addressing the SSE-KMS feature. The S3v4 
signature version has already been implemented in a previous version.


> Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS
> -
>
> Key: NIFI-1769
> URL: https://issues.apache.org/jira/browse/NIFI-1769
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Michiel Moonen
>Priority: Minor
>  Labels: newbie, patch, security
>
> Currently there is no support for SSE-KMS S3 Signature Version 4 
> Authentication. This is necessary for enhanced security features



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2891) The last tab selected/edited should be remembered when configuring a connection

2016-10-12 Thread Andrew Lim (JIRA)

Andrew Lim created NIFI-2891:


 Summary: The last tab selected/edited should be remembered when 
configuring a connection
 Key: NIFI-2891
 URL: https://issues.apache.org/jira/browse/NIFI-2891
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core UI
Affects Versions: 0.7.0, 1.0.0
Reporter: Andrew Lim
Priority: Trivial


When you configure a processor, the Configure Processor window opens with the 
tab that was most recently selected.

This same behavior should occur when configuring a connection.  For example, if 
a user changes the back pressure settings on a connection, the next 
Configuration Connection dialog should open with the Settings tab selected.   
The current behavior is that the Details tab is always the default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2889) Add syslog.ip attribut to ListenSyslog processor

2016-10-12 Thread Bryan Bende (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568619#comment-15568619
 ] 

Bryan Bende commented on NIFI-2889:
---

The sender is coming from calling InetAddress.toString() which says:

{code}
/**
 * Converts this IP address to a {@code String}. The
 * string returned is of the form: hostname / literal IP
 * address.
 *
 * If the host name is unresolved, no reverse name service lookup
 * is performed. The hostname part will be represented by an empty string.
 *
 * @return  a string representation of this IP address.
 */
public String toString() {
String hostName = holder().getHostName();
return ((hostName != null) ? hostName : "")
+ "/" + getHostAddress();
}
{code}

We could still add syslog.ip, but you should be able to easily use an 
UpdateAttribute processor and expression language to get the IP, something like:
{code}
syslog.ip = ${syslog.sender:substringAfter('/')}
{code}

> Add syslog.ip attribut to ListenSyslog processor
> 
>
> Key: NIFI-2889
> URL: https://issues.apache.org/jira/browse/NIFI-2889
> Project: Apache NiFi
>  Issue Type: Wish
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Yohann
>Priority: Minor
>
> Currently, it's not possible to get the IP (v4/v6) address of the syslog 
> server that sent the message.
> So, I would like a {{syslog.ip}} attribut with the IP address of the syslog 
> server. 
> Use case:  
> - Useful for monitoring (with different policy monitoring depending of the IP)
> - The hostname is not always present in the syslog message, so the IP could 
> be used to identify the syslog server that sent the message 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2837) UI - Template upload

2016-10-12 Thread Bryan Bende (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende updated NIFI-2837:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> UI - Template upload
> 
>
> Key: NIFI-2837
> URL: https://issues.apache.org/jira/browse/NIFI-2837
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
> Fix For: 1.1.0
>
>
> When uploading a template, the form submit button should be removed once 
> clicked to prevent accidental re-submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2837) UI - Template upload

2016-10-12 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568690#comment-15568690
 ] 

ASF GitHub Bot commented on NIFI-2837:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1086


> UI - Template upload
> 
>
> Key: NIFI-2837
> URL: https://issues.apache.org/jira/browse/NIFI-2837
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
> Fix For: 1.1.0
>
>
> When uploading a template, the form submit button should be removed once 
> clicked to prevent accidental re-submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2837) UI - Template upload


[ 
https://issues.apache.org/jira/browse/NIFI-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568688#comment-15568688
 ] 

ASF subversion and git services commented on NIFI-2837:
---

Commit a25b651da9da57a20431b9eedcd8a8860aaa4242 in nifi's branch 
refs/heads/master from [~mcgilman]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=a25b651 ]

NIFI-2837: - Immediately hiding the template dialog after clicking the Upload 
button to prevent an accidental re-submission.

This closes #1086.

Signed-off-by: Bryan Bende 


> UI - Template upload
> 
>
> Key: NIFI-2837
> URL: https://issues.apache.org/jira/browse/NIFI-2837
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
> Fix For: 1.1.0
>
>
> When uploading a template, the form submit button should be removed once 
> clicked to prevent accidental re-submission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1086: Hiding the template upload dialog after form submis...

2016-10-12 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1086


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (NIFI-2885) ExecuteScript Should Support View State

2016-10-12 Thread Bryan Bende (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende resolved NIFI-2885.
---
Resolution: Duplicate

> ExecuteScript Should Support View State
> ---
>
> Key: NIFI-2885
> URL: https://issues.apache.org/jira/browse/NIFI-2885
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Bryan Bende
>Priority: Minor
>
> Since scripts executed by ExecuteScript can get access to the state manager 
> and store state, we should mark the processor with the @Stateful annotation 
> so that the View State UI is exposed for this processor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2889) Add syslog.ip attribut to ListenSyslog processor

2016-10-12 Thread Yohann (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567845#comment-15567845
 ] 

Yohann commented on NIFI-2889:
--

Thanks [~bbende],  I didn't tested the  "syslog.sender"  because the 
documentation indicates only the hostname and not the IP address.

However, after a little test with a ListenSyslog processor and AttributesToJSON 
processor:
{code}
$ logger -P 5000 -n 192.168.1.10 -T  test2
{code}

I got the following result :
{code}
{  
   "syslog.hostame":"",
   "syslog.sender":"/192.168.1.10",
   "syslog.body":"yoyo - - [timeQuality tzKnown=\"1\" isSynced=\"1\" 
syncAccuracy=\"114500\"] test2",
   "syslog.timestamp":"2016-10-12T07:36:30.367139+02:00"
}
{code}

So the "syslog.sender" return the correct IP of the sender with the prefix 
{{/}}. 

* What is the expected format of syslog.sender ? Is it {{hostname/ip}} ?  
([ListenSyslog.java#L524|https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenSyslog.java#L524])
* Is it possible to disable the hostname resolution ?  (no DNS PTR request)

 

> Add syslog.ip attribut to ListenSyslog processor
> 
>
> Key: NIFI-2889
> URL: https://issues.apache.org/jira/browse/NIFI-2889
> Project: Apache NiFi
>  Issue Type: Wish
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Yohann
>Priority: Minor
>
> Currently, it's not possible to get the IP (v4/v6) address of the syslog 
> server that sent the message.
> So, I would like a {{syslog.ip}} attribut with the IP address of the syslog 
> server. 
> Use case:  
> - Useful for monitoring (with different policy monitoring depending of the IP)
> - The hostname is not always present in the syslog message, so the IP could 
> be used to identify the syslog server that sent the message 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #239: Nifi 1540 - AWS Kinesis Put Processor

2016-10-12 Thread miquillo

Github user miquillo commented on the issue:

https://github.com/apache/nifi/pull/239
  
@jvwing I kind of overlooked the validator for 1-500 for the BATCH_SIZE 
property, didn't realize it actually captures the constraint of the package 
size. Go with the approach currently implemented.

@mans2singh @jvwing great work! Hope to see this is the next release :)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-1540) Implement AWS Kinesis Put Processor

2016-10-12 Thread James Wing (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Wing updated NIFI-1540:
-
Fix Version/s: 1.1.0

> Implement AWS Kinesis Put Processor
> ---
>
> Key: NIFI-1540
> URL: https://issues.apache.org/jira/browse/NIFI-1540
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 0.5.0
> Environment: all
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: aws, kinesis, put, stream
> Fix For: 1.1.0
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Provide processors to get and put data to AWS Kinesis



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2889) Add syslog.ip attribut to ListenSyslog processor

2016-10-12 Thread Yohann (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569096#comment-15569096
 ] 

Yohann commented on NIFI-2889:
--

I'm Ok with you solution. I close this issue.

> Add syslog.ip attribut to ListenSyslog processor
> 
>
> Key: NIFI-2889
> URL: https://issues.apache.org/jira/browse/NIFI-2889
> Project: Apache NiFi
>  Issue Type: Wish
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Yohann
>Priority: Minor
>
> Currently, it's not possible to get the IP (v4/v6) address of the syslog 
> server that sent the message.
> So, I would like a {{syslog.ip}} attribut with the IP address of the syslog 
> server. 
> Use case:  
> - Useful for monitoring (with different policy monitoring depending of the IP)
> - The hostname is not always present in the syslog message, so the IP could 
> be used to identify the syslog server that sent the message 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2892) Implement AWS Kinesis Stream Get Processor

Stephane Maarek created NIFI-2892:
-

 Summary: Implement AWS Kinesis Stream Get Processor
 Key: NIFI-2892
 URL: https://issues.apache.org/jira/browse/NIFI-2892
 Project: Apache NiFi
  Issue Type: New Feature
Affects Versions: 1.0.0
Reporter: Stephane Maarek


As the Kinesis Put Processor was just implemented in #1540, it would be great 
to have a Kinesis Get Processor.

The main challenges are making sure the SDKs we use for this have a license 
that authorise NiFi to bundle them with the main binaries. 

I also know there are two models to read from a Kinesis Stream (push vs pull), 
and we may want to implement either or both, as part of one or multiple 
processors, for performance purposes. Open for discussions



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2894) PutSQL property Support Fragmented Transactions documentation fix

2016-10-12 Thread Jeff Storck (JIRA)

Jeff Storck created NIFI-2894:
-

 Summary: PutSQL property Support Fragmented Transactions 
documentation fix
 Key: NIFI-2894
 URL: https://issues.apache.org/jira/browse/NIFI-2894
 Project: Apache NiFi
  Issue Type: Task
  Components: Extensions
Affects Versions: 1.0.0
Reporter: Jeff Storck
Priority: Minor


PutSQL property "Support Fragmented Transactions" has an error in its 
documentation, with the proposed edit:

bq. If the fragment.count value is greater than 1, the Processor will not 
process any FlowFile -will-+with+ that fragment.identifier until all are 
available;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2893) Missing or wrong API doc


 [ 
https://issues.apache.org/jira/browse/NIFI-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephane Maarek updated NIFI-2893:
--
Summary: Missing or wrong API doc  (was: Missing API doc)

> Missing or wrong API doc
> 
>
> Key: NIFI-2893
> URL: https://issues.apache.org/jira/browse/NIFI-2893
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Stephane Maarek
>Priority: Minor
>
> At:
> https://nifi.apache.org/docs/nifi-docs/rest-api/
> We're missing the GET  /process-groups/{id}/controller-services
> To list all the controller services within process groups. Only POST 
> currently exists



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2893) Missing API doc


 [ 
https://issues.apache.org/jira/browse/NIFI-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephane Maarek updated NIFI-2893:
--
Summary: Missing API doc  (was: Missing API doc for controller services in 
process group)

> Missing API doc
> ---
>
> Key: NIFI-2893
> URL: https://issues.apache.org/jira/browse/NIFI-2893
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Stephane Maarek
>Priority: Minor
>
> At:
> https://nifi.apache.org/docs/nifi-docs/rest-api/
> We're missing the GET  /process-groups/{id}/controller-services
> To list all the controller services within process groups. Only POST 
> currently exists



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2894) PutSQL property Support Fragmented Transactions documentation fix

2016-10-12 Thread Jeff Storck (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Storck updated NIFI-2894:
--
Fix Version/s: 1.1.0

> PutSQL property Support Fragmented Transactions documentation fix
> -
>
> Key: NIFI-2894
> URL: https://issues.apache.org/jira/browse/NIFI-2894
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Jeff Storck
>Priority: Minor
>  Labels: documentation
> Fix For: 1.1.0
>
>
> PutSQL property "Support Fragmented Transactions" has an error in its 
> documentation, with the proposed edit:
> bq. If the fragment.count value is greater than 1, the Processor will not 
> process any FlowFile -will-+with+ that fragment.identifier until all are 
> available;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2893) Missing or wrong API doc


 [ 
https://issues.apache.org/jira/browse/NIFI-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephane Maarek updated NIFI-2893:
--
Description: 
At:
https://nifi.apache.org/docs/nifi-docs/rest-api/

Missing:
- GET  /process-groups/{id}/controller-services


Erroneous:
- POST /process-groups/{id}/templates/upload
> should give a 201 in case of created, returns Location in header so we 
can get the ID out of it
> it'd be good to have a description of the returned xml (nodes, etc)

  was:
At:
https://nifi.apache.org/docs/nifi-docs/rest-api/

Missing:
- GET  /process-groups/{id}/controller-services

Erroneous:
- POST /process-groups/{id}/templates/upload
> should give a 201 in case of created, returns Location in header so we 
can get the ID out of it
> it'd be good to have a description of the returned xml (nodes, etc)


> Missing or wrong API doc
> 
>
> Key: NIFI-2893
> URL: https://issues.apache.org/jira/browse/NIFI-2893
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Stephane Maarek
>Priority: Minor
>
> At:
> https://nifi.apache.org/docs/nifi-docs/rest-api/
> Missing:
> - GET  /process-groups/{id}/controller-services
> Erroneous:
> - POST /process-groups/{id}/templates/upload
> > should give a 201 in case of created, returns Location in header so we 
> can get the ID out of it
> > it'd be good to have a description of the returned xml (nodes, etc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2893) Missing or wrong API doc


 [ 
https://issues.apache.org/jira/browse/NIFI-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephane Maarek updated NIFI-2893:
--
Description: 
At:
https://nifi.apache.org/docs/nifi-docs/rest-api/

Missing:
- GET  /process-groups/{id}/controller-services

Erroneous:
- POST /process-groups/{id}/templates/upload
> should give a 201 in case of created, returns Location in header so we 
can get the ID out of it
> it'd be good to have a description of the returned xml (nodes, etc)

  was:
At:
https://nifi.apache.org/docs/nifi-docs/rest-api/

We're missing the GET  /process-groups/{id}/controller-services
To list all the controller services within process groups. Only POST currently 
exists



> Missing or wrong API doc
> 
>
> Key: NIFI-2893
> URL: https://issues.apache.org/jira/browse/NIFI-2893
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Stephane Maarek
>Priority: Minor
>
> At:
> https://nifi.apache.org/docs/nifi-docs/rest-api/
> Missing:
> - GET  /process-groups/{id}/controller-services
> Erroneous:
> - POST /process-groups/{id}/templates/upload
> > should give a 201 in case of created, returns Location in header so we 
> can get the ID out of it
> > it'd be good to have a description of the returned xml (nodes, etc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2799) AWS Credentials for Assume Role Need Proxy

2016-10-12 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570349#comment-15570349
 ] 

ASF GitHub Bot commented on NIFI-2799:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1112


> AWS Credentials for Assume Role Need Proxy
> --
>
> Key: NIFI-2799
> URL: https://issues.apache.org/jira/browse/NIFI-2799
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Keren Tseytlin
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.1.0
>
>
> As a user of Nifi, when I want to enable cross account fetching of S3 objects 
> I need the proxy variables to be set in order to generate temporary AWS 
> tokens for STS:AssumeRole.
> Within some enterprise environments, it is necessary to set the proxy 
> variables prior to running AssumeRole methods. Without this being set, the 
> machine in VPC A times out on generating temporary keys and is unable to 
> assume a role as a machine in VPC B. 
> This ticket arose from this conversation: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-Cross-Account-Download-With-A-Profile-Flag-td13232.html#a13252
> Goal: There are files stored in an S3 bucket in VPC B. My Nifi machines are 
> in VPC A. I want Nifi to be able to get those files from VPC B. VPC A and VPC 
> B need to be communicating in the FetchS3Object component.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1112: NIFI-2799: AWS Credentials for Assume Role Need Pro...

2016-10-12 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1112


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2799) AWS Credentials for Assume Role Need Proxy


[ 
https://issues.apache.org/jira/browse/NIFI-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570348#comment-15570348
 ] 

ASF subversion and git services commented on NIFI-2799:
---

Commit 6dc2f14198bb475ace8e762e348ca5294400cccb in nifi's branch 
refs/heads/master from [~ktseytlin]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=6dc2f14 ]

NIFI-2799 AWS Assume Role Credentials With Proxy

This closes #1112.

Signed-off-by: James Wing 


> AWS Credentials for Assume Role Need Proxy
> --
>
> Key: NIFI-2799
> URL: https://issues.apache.org/jira/browse/NIFI-2799
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Keren Tseytlin
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.1.0
>
>
> As a user of Nifi, when I want to enable cross account fetching of S3 objects 
> I need the proxy variables to be set in order to generate temporary AWS 
> tokens for STS:AssumeRole.
> Within some enterprise environments, it is necessary to set the proxy 
> variables prior to running AssumeRole methods. Without this being set, the 
> machine in VPC A times out on generating temporary keys and is unable to 
> assume a role as a machine in VPC B. 
> This ticket arose from this conversation: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-Cross-Account-Download-With-A-Profile-Flag-td13232.html#a13252
> Goal: There are files stored in an S3 bucket in VPC B. My Nifi machines are 
> in VPC A. I want Nifi to be able to get those files from VPC B. VPC A and VPC 
> B need to be communicating in the FetchS3Object component.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2893) Missing API doc for controller services in process group


 [ 
https://issues.apache.org/jira/browse/NIFI-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephane Maarek updated NIFI-2893:
--
Description: 
At:
https://nifi.apache.org/docs/nifi-docs/rest-api/

We're missing the GET  /process-groups/{id}/controller-services
To list all the controller services within process groups. Only POST currently 
exists


  was:
At:
https://nifi.apache.org/docs/nifi-docs/rest-api/

We're missing the GET  /process-groups/{id}/controller-services
To list all the controller services within process groups. Only PUT currently 
exists



> Missing API doc for controller services in process group
> 
>
> Key: NIFI-2893
> URL: https://issues.apache.org/jira/browse/NIFI-2893
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Stephane Maarek
>Priority: Minor
>
> At:
> https://nifi.apache.org/docs/nifi-docs/rest-api/
> We're missing the GET  /process-groups/{id}/controller-services
> To list all the controller services within process groups. Only POST 
> currently exists



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1112: NIFI-2799: AWS Credentials for Assume Role Need Proxy

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/1112
  
@ktseytlin Thanks for the updates.  The code looks good, and the improved 
Assume Role worked well in my testing with and without a proxy.  I will merge 
shortly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-2893) Missing API doc for controller services in process group

Stephane Maarek created NIFI-2893:
-

 Summary: Missing API doc for controller services in process group
 Key: NIFI-2893
 URL: https://issues.apache.org/jira/browse/NIFI-2893
 Project: Apache NiFi
  Issue Type: Bug
  Components: Documentation & Website
Affects Versions: 1.0.0
Reporter: Stephane Maarek
Priority: Minor


At:
https://nifi.apache.org/docs/nifi-docs/rest-api/

We're missing the GET  /process-groups/{id}/controller-services
To list all the controller services within process groups. Only PUT currently 
exists




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2799) AWS Credentials for Assume Role Need Proxy


[ 
https://issues.apache.org/jira/browse/NIFI-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570343#comment-15570343
 ] 

ASF GitHub Bot commented on NIFI-2799:
--

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/1112
  
@ktseytlin Thanks for the updates.  The code looks good, and the improved 
Assume Role worked well in my testing with and without a proxy.  I will merge 
shortly.


> AWS Credentials for Assume Role Need Proxy
> --
>
> Key: NIFI-2799
> URL: https://issues.apache.org/jira/browse/NIFI-2799
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Keren Tseytlin
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.1.0
>
>
> As a user of Nifi, when I want to enable cross account fetching of S3 objects 
> I need the proxy variables to be set in order to generate temporary AWS 
> tokens for STS:AssumeRole.
> Within some enterprise environments, it is necessary to set the proxy 
> variables prior to running AssumeRole methods. Without this being set, the 
> machine in VPC A times out on generating temporary keys and is unable to 
> assume a role as a machine in VPC B. 
> This ticket arose from this conversation: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-Cross-Account-Download-With-A-Profile-Flag-td13232.html#a13252
> Goal: There are files stored in an S3 bucket in VPC B. My Nifi machines are 
> in VPC A. I want Nifi to be able to get those files from VPC B. VPC A and VPC 
> B need to be communicating in the FetchS3Object component.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #239: Nifi 1540 - AWS Kinesis Put Processor

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/239
  
@simplesteph - Please open a new ticket for a GetKinesis processor, I'll 
resolve the current ticket.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (NIFI-2889) Add syslog.ip attribut to ListenSyslog processor

2016-10-12 Thread Yohann (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yohann resolved NIFI-2889.
--
Resolution: Workaround

> Add syslog.ip attribut to ListenSyslog processor
> 
>
> Key: NIFI-2889
> URL: https://issues.apache.org/jira/browse/NIFI-2889
> Project: Apache NiFi
>  Issue Type: Wish
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Yohann
>Priority: Minor
>
> Currently, it's not possible to get the IP (v4/v6) address of the syslog 
> server that sent the message.
> So, I would like a {{syslog.ip}} attribut with the IP address of the syslog 
> server. 
> Use case:  
> - Useful for monitoring (with different policy monitoring depending of the IP)
> - The hostname is not always present in the syslog message, so the IP could 
> be used to identify the syslog server that sent the message 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2866) The Initial Max Value of QueryDatabaseTable won't be case sensitive.


[ 
https://issues.apache.org/jira/browse/NIFI-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569388#comment-15569388
 ] 

ASF GitHub Bot commented on NIFI-2866:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/1101
  
Can you add a unit test for this? One option is to update the 
testInitialMaxValue() test to change the name of the "created_on" column to 
uppercase, as well as the max-value column name and the 
initial.maxvalue.CREATED_ON attribute. That should be enough to illustrate the 
problem before applying your change, then the test passes after your change.


> The Initial Max Value of QueryDatabaseTable won't be case sensitive.
> 
>
> Key: NIFI-2866
> URL: https://issues.apache.org/jira/browse/NIFI-2866
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.1.0
>Reporter: Byunghwa Yun
>Priority: Minor
>
> Now, the Initial Max Value of QueryDatabaseTable is allowed the only 
> lowercase so I think to change the below code, it added .toLowerCase() method.
> for(final Map.Entry maxProp : 
> maxValueProperties.entrySet()){
> if 
> (!statePropertyMap.containsKey(maxProp.getKey().toLowerCase())) {
> statePropertyMap.put(maxProp.getKey().toLowerCase(), 
> maxProp.getValue());
> }
> }
> When I set the property that is initial.maxvalue.UPDATE_TIME, 
> QueryDatabaseTable doesn't have WHERE clause with UPDATE_TIME.
> Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1101: NIFI-2866 The Initial Max Value of QueryDatabaseTable won'...

2016-10-12 Thread mattyb149

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/1101
  
Can you add a unit test for this? One option is to update the 
testInitialMaxValue() test to change the name of the "created_on" column to 
uppercase, as well as the max-value column name and the 
initial.maxvalue.CREATED_ON attribute. That should be enough to illustrate the 
problem before applying your change, then the test passes after your change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2862) Add Stateful annotation to scripting processors


[ 
https://issues.apache.org/jira/browse/NIFI-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569305#comment-15569305
 ] 

Matt Burgess commented on NIFI-2862:


+1 LGTM, built and ran, verified the View State option is enabled on the 
scripting processors, and the description is accurately displayed.

Merging to master, thank you for the contribution!

> Add Stateful annotation to scripting processors
> ---
>
> Key: NIFI-2862
> URL: https://issues.apache.org/jira/browse/NIFI-2862
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Peter Mitchell
>Priority: Trivial
> Fix For: 1.1.0
>
> Attachments: NIFI-2682_2.patch, nifi-2862.patch
>
>
> Scripting processors are able to access the state management apis. It would 
> be useful to have the Stateful annotation added to these processors to enable 
> DFMs to clear any state that may be stored by a script.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2862) Add Stateful annotation to scripting processors


 [ 
https://issues.apache.org/jira/browse/NIFI-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-2862:
---
Attachment: NIFI-2682_2.patch

Added additional text and uppercased "APIs". Added patch to Jira case

> Add Stateful annotation to scripting processors
> ---
>
> Key: NIFI-2862
> URL: https://issues.apache.org/jira/browse/NIFI-2862
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Peter Mitchell
>Priority: Trivial
> Fix For: 1.1.0
>
> Attachments: NIFI-2682_2.patch, nifi-2862.patch
>
>
> Scripting processors are able to access the state management apis. It would 
> be useful to have the Stateful annotation added to these processors to enable 
> DFMs to clear any state that may be stored by a script.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2862) Add Stateful annotation to scripting processors


 [ 
https://issues.apache.org/jira/browse/NIFI-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-2862:
---
Fix Version/s: 1.1.0

> Add Stateful annotation to scripting processors
> ---
>
> Key: NIFI-2862
> URL: https://issues.apache.org/jira/browse/NIFI-2862
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Peter Mitchell
>Priority: Trivial
> Fix For: 1.1.0
>
> Attachments: NIFI-2682_2.patch, nifi-2862.patch
>
>
> Scripting processors are able to access the state management apis. It would 
> be useful to have the Stateful annotation added to these processors to enable 
> DFMs to clear any state that may be stored by a script.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2862) Add Stateful annotation to scripting processors

2016-10-12 Thread ASF subversion and git services (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-2862:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Add Stateful annotation to scripting processors
> ---
>
> Key: NIFI-2862
> URL: https://issues.apache.org/jira/browse/NIFI-2862
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Peter Mitchell
>Priority: Trivial
> Fix For: 1.1.0
>
> Attachments: NIFI-2682_2.patch, nifi-2862.patch
>
>
> Scripting processors are able to access the state management apis. It would 
> be useful to have the Stateful annotation added to these processors to enable 
> DFMs to clear any state that may be stored by a script.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1540) Implement AWS Kinesis Put Processor

2016-10-12 Thread James Wing (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Wing updated NIFI-1540:
-
Resolution: Resolved
Status: Resolved  (was: Patch Available)

> Implement AWS Kinesis Put Processor
> ---
>
> Key: NIFI-1540
> URL: https://issues.apache.org/jira/browse/NIFI-1540
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 0.5.0
> Environment: all
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: aws, kinesis, put, stream
> Fix For: 1.1.0
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Provide processors to get and put data to AWS Kinesis



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #239: Nifi 1540 - AWS Kinesis Put Processor

2016-10-12 Thread joewitt

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/239
  
Nice job on this @mans2singh @jvwing @simplesteph @apiri !  Good discussion 
of various tradeoffs.  Handled the license implications well (I've been double 
checking and all appear to be ASLv2!).  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2682) Encrypt toolkit test fails for European timezone


[ 
https://issues.apache.org/jira/browse/NIFI-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569309#comment-15569309
 ] 

ASF subversion and git services commented on NIFI-2682:
---

Commit 893fed794c20f8f9ce87e7e60f5ad9722a67b3bd in nifi's branch 
refs/heads/master from [~petmit]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=893fed7 ]

Add Stateful annotation to scripting processors

Signed-off-by: Matt Burgess 

NIFI-2682: Added text to Stateful description for AbstractScriptProcessor


> Encrypt toolkit test fails for European timezone
> 
>
> Key: NIFI-2682
> URL: https://issues.apache.org/jira/browse/NIFI-2682
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.0.0
> Environment: Apache Maven 3.3.9 
> (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T17:41:47+01:00)
> Java version: 1.8.0_77, vendor: Oracle Corporation
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: "mac"
> CEST time zone
>Reporter: Pierre Villard
>Priority: Minor
>
> I have the following test failure when building from sources:
> {noformat}
> ---
>  T E S T S
> ---
> Running org.apache.nifi.properties.ConfigEncryptionToolTest
> Tests run: 47, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 2.397 sec 
> <<< FAILURE! - in org.apache.nifi.properties.ConfigEncryptionToolTest
> testShouldSerializeNiFiProperties(org.apache.nifi.properties.ConfigEncryptionToolTest)
>   Time elapsed: 0.011 sec  <<< FAILURE!
> org.codehaus.groovy.runtime.powerassert.PowerAssertionError: assert 
> lines.first() =~ datePattern
>    | |   |  |
>    | |   |  ^#\w{3} \w{3} \d{2} \d{2}:\d{2}:\d{2} 
> \w{3}([\-+]\d{2}:\d{2})? \d{4}$
>    | |   java.util.regex.Matcher[pattern=^#\w{3} \w{3} \d{2} 
> \d{2}:\d{2}:\d{2} \w{3}([\-+]\d{2}:\d{2})? \d{4}$ region=0,30 lastmatch=]
>    | #Sat Aug 27 17:40:00 CEST 2016
>    [#Sat Aug 27 17:40:00 CEST 2016, key=value, key2=value2]
>   at 
> org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:402)
>   at 
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:650)
>   at 
> org.apache.nifi.properties.ConfigEncryptionToolTest.testShouldSerializeNiFiProperties(ConfigEncryptionToolTest.groovy:978)
> Results :
> Failed tests: 
>   ConfigEncryptionToolTest.testShouldSerializeNiFiProperties:978 assert 
> lines.first() =~ datePattern
>    | |   |  |
>    | |   |  ^#\w{3} \w{3} \d{2} \d{2}:\d{2}:\d{2} 
> \w{3}([\-+]\d{2}:\d{2})? \d{4}$
>    | |   java.util.regex.Matcher[pattern=^#\w{3} \w{3} \d{2} 
> \d{2}:\d{2}:\d{2} \w{3}([\-+]\d{2}:\d{2})? \d{4}$ region=0,30 lastmatch=]
>    | #Sat Aug 27 17:40:00 CEST 2016
>    [#Sat Aug 27 17:40:00 CEST 2016, key=value, key2=value2]
> Tests run: 47, Failures: 1, Errors: 0, Skipped: 2
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi-minifi-cpp issue #18: MINIFI-34 - attempt to progress the CMake environ...

2016-10-12 Thread trixpan

Github user trixpan commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/18
  
@apiri should we merge this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2801) Add information about relevant version to *Kafka processor documentation

2016-10-12 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570590#comment-15570590
 ] 

ASF subversion and git services commented on NIFI-2801:
---

Commit 99874055bd649895dd4ddc68e294998929cbb001 in nifi's branch refs/heads/0.x 
from [~andrewmlim]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=9987405 ]

NIFI-2801 Fixed checkstyle line length issue in PutKafka. (+1 squashed commit)
Squashed commits:
[6e8f160] NIFI-2801 Edited Kafka processor documentation to explicitly state 
which Kafka versions supported by each processor.

This closes #1124.

Signed-off-by: Andy LoPresto 


> Add information about relevant version to *Kafka processor documentation
> 
>
> Key: NIFI-2801
> URL: https://issues.apache.org/jira/browse/NIFI-2801
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andy LoPresto
>Assignee: Andrew Lim
>Priority: Minor
>  Labels: Beginner
Documentation
Kafka
> Fix For: 1.1.0, 0.8.0
>
>
> A frequent obstacle for new users is the various processors for communicating 
> with Kafka. Due to incompatibilities between different versions of Kafka, 
> there are currently 6 processors (3 to push data and 3 to pull data), and 
> each "pair" targets a specific version. We should add text to the 
> documentation/description of each to clarify explicitly which version of 
> Kafka the processor targets, and what its "complementary" processor is named. 
> * <=0.8 -- PutKafka/GetKafka
> * 0.9 -- PublishKafka/ConsumeKafka
> * 0.10 -- PublishKafka_0_10/ConsumeKafka_0_10



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2801) Add information about relevant version to *Kafka processor documentation

2016-10-12 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570591#comment-15570591
 ] 

ASF subversion and git services commented on NIFI-2801:
---

Commit 99874055bd649895dd4ddc68e294998929cbb001 in nifi's branch refs/heads/0.x 
from [~andrewmlim]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=9987405 ]

NIFI-2801 Fixed checkstyle line length issue in PutKafka. (+1 squashed commit)
Squashed commits:
[6e8f160] NIFI-2801 Edited Kafka processor documentation to explicitly state 
which Kafka versions supported by each processor.

This closes #1124.

Signed-off-by: Andy LoPresto 


> Add information about relevant version to *Kafka processor documentation
> 
>
> Key: NIFI-2801
> URL: https://issues.apache.org/jira/browse/NIFI-2801
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andy LoPresto
>Assignee: Andrew Lim
>Priority: Minor
>  Labels: Beginner
Documentation
Kafka
> Fix For: 1.1.0, 0.8.0
>
>
> A frequent obstacle for new users is the various processors for communicating 
> with Kafka. Due to incompatibilities between different versions of Kafka, 
> there are currently 6 processors (3 to push data and 3 to pull data), and 
> each "pair" targets a specific version. We should add text to the 
> documentation/description of each to clarify explicitly which version of 
> Kafka the processor targets, and what its "complementary" processor is named. 
> * <=0.8 -- PutKafka/GetKafka
> * 0.9 -- PublishKafka/ConsumeKafka
> * 0.10 -- PublishKafka_0_10/ConsumeKafka_0_10



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2853) Improve ListHDFS state tracking

2016-10-12 Thread Joseph Witt (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570581#comment-15570581
 ] 

Joseph Witt commented on NIFI-2853:
---

If very clearly documented I could see how ListHDFS could be allowed to run on 
multiple nodes if configured to look at unique directories but it also seems to 
be a potentially unnecessary complexity.

The idea for ListHDFS was to something very lightweight from a single node and 
the work could then be farmed out for the heavier listing across the cluster 
which would be the actual FetchHDFS calls.

By allowing multiple nodes to execute ListHDFS at once we have to have some way 
to namespace the state and while the directory being listed would do it this 
also means we could end up needing to store an arbitrarily large number of 
directories.  In the single node case regardless of how many directories we're 
pulling from it won't matter because a single value for timestamp of listing 
and emitting is sufficient for all of them (just look for anything in any 
matching directory that has changed since that time).

> Improve ListHDFS state tracking
> ---
>
> Key: NIFI-2853
> URL: https://issues.apache.org/jira/browse/NIFI-2853
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Bryan Bende
>Priority: Minor
>
> Currently ListHDFS tracks two properties in state management, 
> "listing.timestamp" and "emitted.timestamp". In the 1.0.0 release, the 
> directory property now supports expression language which means the directory 
> being listed could dynamically change on any execution of the processor. 
> The processor should be changed to store state specific to the directory that 
> was listed, for example "listing.timestamp.dir1" and "emitted.timestamp.dir1".
> This would also help in a clustered scenario... currently ListHDFS has to be 
> run on primary node only, otherwise each node will be overwriting each others 
> state and producing unexpected results. With the above improvement, if the 
> directory evaluated to a unique path for each node, it would store the state 
> of each of those path separately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2801) Add information about relevant version to *Kafka processor documentation


[ 
https://issues.apache.org/jira/browse/NIFI-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570617#comment-15570617
 ] 

ASF GitHub Bot commented on NIFI-2801:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/1124
  
Reviewed this, fix a line-length error in `PutKafka`, and merged to Apache 
canonical repo 
[9cbb001](https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=99874055bd649895dd4ddc68e294998929cbb001).
 The GitHub repository hasn't synced the change, so the PR remains open. 


> Add information about relevant version to *Kafka processor documentation
> 
>
> Key: NIFI-2801
> URL: https://issues.apache.org/jira/browse/NIFI-2801
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andy LoPresto
>Assignee: Andrew Lim
>Priority: Minor
>  Labels: Beginner
Documentation
Kafka
> Fix For: 1.1.0, 0.8.0
>
>
> A frequent obstacle for new users is the various processors for communicating 
> with Kafka. Due to incompatibilities between different versions of Kafka, 
> there are currently 6 processors (3 to push data and 3 to pull data), and 
> each "pair" targets a specific version. We should add text to the 
> documentation/description of each to clarify explicitly which version of 
> Kafka the processor targets, and what its "complementary" processor is named. 
> * <=0.8 -- PutKafka/GetKafka
> * 0.9 -- PublishKafka/ConsumeKafka
> * 0.10 -- PublishKafka_0_10/ConsumeKafka_0_10



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1124: NIFI-2801 Edited Kafka processor documentation to explicit...

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/1124
  
Reviewed this, fix a line-length error in `PutKafka`, and merged to Apache 
canonical repo 
[9cbb001](https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=99874055bd649895dd4ddc68e294998929cbb001).
 The GitHub repository hasn't synced the change, so the PR remains open. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2861) ControlRate should accept more than one flow file per execution


[ 
https://issues.apache.org/jira/browse/NIFI-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570913#comment-15570913
 ] 

ASF GitHub Bot commented on NIFI-2861:
--

GitHub user jskora opened a pull request:

https://github.com/apache/nifi/pull/1127

NIFI-2861 ControlRate should accept more than one flow file per execution 
(1.x)

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.

* Support multiple files per onTrigger call.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jskora/nifi NIFI-2861-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1127.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1127


commit d66dd6d0f59784da14a26fce50542224e1cf4b07
Author: Joe Skora 
Date:   2016-10-13T05:14:23Z

NIFI-2861 ControlRate should accept more than one flow file per execution

* Support multiple files per onTrigger call.




> ControlRate should accept more than one flow file per execution
> ---
>
> Key: NIFI-2861
> URL: https://issues.apache.org/jira/browse/NIFI-2861
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Joe Skora
>Assignee: Joe Skora
>
> The {{ControlRate}} processor implements a {{FlowFileFilter}} that returns 
> the {{FlowFileFilter.ACCEPT_AND_TERMINATE}} result if the {{FlowFile}} fits 
> with the rate limit, affectively limiting it to one {{FlowFile}} per 
> {{ConrolRate.onTrigger()}} invocation.  This is a significant bottleneck when 
> processing very large quantities of small files making it unlikely to hit the 
> rate limits.
> It should allow multiple files, perhaps with a configurable maximum, per 
> {{ControlRate.onTrigger()}} invocation by issuing the 
> {{FlowFileFilter.ACCEPT_AND_CONTINUE}} result until the limits are reached.  
> In a preliminary test this eliminated the bottleneck.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1127: NIFI-2861 ControlRate should accept more than one f...

2016-10-12 Thread jskora

GitHub user jskora opened a pull request:

https://github.com/apache/nifi/pull/1127

NIFI-2861 ControlRate should accept more than one flow file per execution 
(1.x)

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.

* Support multiple files per onTrigger call.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jskora/nifi NIFI-2861-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1127.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1127


commit d66dd6d0f59784da14a26fce50542224e1cf4b07
Author: Joe Skora 
Date:   2016-10-13T05:14:23Z

NIFI-2861 ControlRate should accept more than one flow file per execution

* Support multiple files per onTrigger call.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-2895) Controller Services attached to Process Groups are not saved in a template