date:20160223

[GitHub] nifi pull request: Nifi 1495 - AWS Kinesis Firehose

2016-02-23 Thread mans2singh

Github user mans2singh commented on the pull request:

https://github.com/apache/nifi/pull/213#issuecomment-188062568
  
Corrected title of pull request.  Please let me know if there is anything 
else required.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread mans2singh

Github user mans2singh commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53893312
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/firehose/ITPutKinesisFirehose.java
 ---
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import java.util.List;
+
+import org.apache.nifi.processors.aws.s3.FetchS3Object;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * This test contains both unit and integration test (integration tests 
are ignored by default)
+ */
+public class ITPutKinesisFirehose {
+
+private TestRunner runner;
+protected final static String CREDENTIALS_FILE = 
System.getProperty("user.home") + "/aws-credentials.properties";
+
+@Before
+public void setUp() throws Exception {
+runner = TestRunners.newTestRunner(PutKinesisFirehose.class);
+runner.setProperty(PutKinesisFirehose.ACCESS_KEY, "abcd");
+runner.setProperty(PutKinesisFirehose.SECRET_KEY, "secret key");
+
runner.setProperty(PutKinesisFirehose.KINESIS_FIREHOSE_DELIVERY_STREAM_NAME, 
"deliveryName");
+runner.assertValid();
+}
+
+@After
+public void tearDown() throws Exception {
+runner = null;
+}
+
+@Test
+public void testCustomValidateBatchSize1Valid() {
+runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "1");
+runner.assertValid();
+}
+
+@Test
+public void testCustomValidateBatchSize500Valid() {
+runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "500");
+runner.assertValid();
+}
+@Test
+public void testCustomValidateBatchSize501InValid() {
+runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "501");
+runner.assertNotValid();
+}
+
+@Test
+public void testCustomValidateBufferSize1Valid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_SIZE, "1");
+runner.assertValid();
+}
+
+@Test
+public void testCustomValidateBufferSize128Valid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_SIZE, "128");
+runner.assertValid();
+}
+@Test
+public void testCustomValidateBufferSize129InValid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_SIZE, "129");
+runner.assertNotValid();
+}
+
+@Test
+public void testCustomValidateBufferInterval900Valid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_INTERVAL, "900");
+runner.assertValid();
+}
+
+@Test
+public void testCustomValidateBufferInterval60Valid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_INTERVAL, "60");
+runner.assertValid();
+}
+
+@Test
+public void testCustomValidateBufferInterval901InValid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_INTERVAL, "901");
+runner.assertNotValid();
+}
+
+@Test
+public void testCustomValidateBufferInterval59InValid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_INTERVAL, "59");
+runner.assertNotValid();
+}
+
+/**
+ * Comment out ignore for integration tests (requires creds files)
+ */
+@Test
+@Ignore
--- End diff --

Removed ignore annotation for IT tests


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not wo

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread mans2singh

Github user mans2singh commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53893291
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/firehose/ITPutKinesisFirehose.java
 ---
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import java.util.List;
+
+import org.apache.nifi.processors.aws.s3.FetchS3Object;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * This test contains both unit and integration test (integration tests 
are ignored by default)
+ */
+public class ITPutKinesisFirehose {
--- End diff --

Moved unit test to separate class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread mans2singh

Github user mans2singh commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53893255
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/firehose/PutKinesisFirehose.java
 ---
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import com.amazonaws.services.kinesisfirehose.AmazonKinesisFirehoseClient;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchRequest;
+import 
com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResponseEntry;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResult;
+import com.amazonaws.services.kinesisfirehose.model.Record;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"amazon", "aws", "firehose", "kinesis", "put", "stream"})
+@CapabilityDescription("Sends the contents to a specified Amazon Kinesis 
Firehose. "
++ "In order to send data to firehose, the firehose delivery stream 
name has to be specified.")
+@WritesAttributes({
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.message", 
description = "Error message on posting message to AWS Kinesis Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.code", 
description = "Error code for the message when posting to AWS Kinesis 
Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.record.id", 
description = "Record id of the message posted to Kinesis Firehose")})
+public class PutKinesisFirehose extends AbstractKinesisFirehoseProcessor {
+
+/**
+ * Kinesis put record response error message
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_MESSAGE = 
"aws.kinesis.firehose.error.message";
+
+/**
+ * Kinesis put record response error code
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_CODE = 
"aws.kinesis.firehose.error.code";
+
+/**
+ * Kinesis put record response record id
+ */
+public static final String AWS_KINESIS_FIREHOSE_RECORD_ID = 
"aws.kinesis.firehose.record.id";
+
+public static final List properties = 
Collections.unmodifiableList(
+Arrays.asList(KINESIS_FIREHOSE_DELIVERY_STREAM_NAME, 
MAX_BUFFER_INTERVAL,
+  MAX_BUFFER_SIZE, BATCH_SIZE, REGION, ACCESS_KEY, 
SECRET_KEY, CREDENTIALS_FILE, AWS_CREDENTIALS_PROVIDER_SERVICE, TIMEOUT,
--- End diff --

I've removed them (these are configuration params which I mistakenly left 
in the client code).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infras

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread mans2singh

Github user mans2singh commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53893277
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/firehose/PutKinesisFirehose.java
 ---
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import com.amazonaws.services.kinesisfirehose.AmazonKinesisFirehoseClient;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchRequest;
+import 
com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResponseEntry;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResult;
+import com.amazonaws.services.kinesisfirehose.model.Record;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"amazon", "aws", "firehose", "kinesis", "put", "stream"})
+@CapabilityDescription("Sends the contents to a specified Amazon Kinesis 
Firehose. "
++ "In order to send data to firehose, the firehose delivery stream 
name has to be specified.")
+@WritesAttributes({
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.message", 
description = "Error message on posting message to AWS Kinesis Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.code", 
description = "Error code for the message when posting to AWS Kinesis 
Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.record.id", 
description = "Record id of the message posted to Kinesis Firehose")})
+public class PutKinesisFirehose extends AbstractKinesisFirehoseProcessor {
+
+/**
+ * Kinesis put record response error message
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_MESSAGE = 
"aws.kinesis.firehose.error.message";
+
+/**
+ * Kinesis put record response error code
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_CODE = 
"aws.kinesis.firehose.error.code";
+
+/**
+ * Kinesis put record response record id
+ */
+public static final String AWS_KINESIS_FIREHOSE_RECORD_ID = 
"aws.kinesis.firehose.record.id";
+
+public static final List properties = 
Collections.unmodifiableList(
+Arrays.asList(KINESIS_FIREHOSE_DELIVERY_STREAM_NAME, 
MAX_BUFFER_INTERVAL,
+  MAX_BUFFER_SIZE, BATCH_SIZE, REGION, ACCESS_KEY, 
SECRET_KEY, CREDENTIALS_FILE, AWS_CREDENTIALS_PROVIDER_SERVICE, TIMEOUT,
+  PROXY_HOST,PROXY_HOST_PORT));
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@Override
+protected Collection customValidate(final 
ValidationContext validationContext) {
+final List problems = new 
ArrayList<>(super.customValidate(validationContext));

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread mans2singh

Github user mans2singh commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53893236
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/firehose/PutKinesisFirehose.java
 ---
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import com.amazonaws.services.kinesisfirehose.AmazonKinesisFirehoseClient;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchRequest;
+import 
com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResponseEntry;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResult;
+import com.amazonaws.services.kinesisfirehose.model.Record;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"amazon", "aws", "firehose", "kinesis", "put", "stream"})
+@CapabilityDescription("Sends the contents to a specified Amazon Kinesis 
Firehose. "
++ "In order to send data to firehose, the firehose delivery stream 
name has to be specified.")
+@WritesAttributes({
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.message", 
description = "Error message on posting message to AWS Kinesis Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.code", 
description = "Error code for the message when posting to AWS Kinesis 
Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.record.id", 
description = "Record id of the message posted to Kinesis Firehose")})
+public class PutKinesisFirehose extends AbstractKinesisFirehoseProcessor {
+
+/**
+ * Kinesis put record response error message
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_MESSAGE = 
"aws.kinesis.firehose.error.message";
+
+/**
+ * Kinesis put record response error code
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_CODE = 
"aws.kinesis.firehose.error.code";
+
+/**
+ * Kinesis put record response record id
+ */
+public static final String AWS_KINESIS_FIREHOSE_RECORD_ID = 
"aws.kinesis.firehose.record.id";
+
+public static final List properties = 
Collections.unmodifiableList(
+Arrays.asList(KINESIS_FIREHOSE_DELIVERY_STREAM_NAME, 
MAX_BUFFER_INTERVAL,
+  MAX_BUFFER_SIZE, BATCH_SIZE, REGION, ACCESS_KEY, 
SECRET_KEY, CREDENTIALS_FILE, AWS_CREDENTIALS_PROVIDER_SERVICE, TIMEOUT,
+  PROXY_HOST,PROXY_HOST_PORT));
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@Override
+protected Collection customValidate(final 
ValidationContext validationContext) {
--- End diff --

I've removed the custom validator code


---
If your project is

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread mans2singh

Github user mans2singh commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53893209
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/firehose/AbstractKinesisFirehoseProcessor.java
 ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor;
+
+import com.amazonaws.ClientConfiguration;
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.services.kinesisfirehose.AmazonKinesisFirehoseClient;
+
+/**
+ * This class provides processor the base class for kinesis firehose
+ */
+public abstract class AbstractKinesisFirehoseProcessor extends 
AbstractAWSCredentialsProviderProcessor {
+
+public static final PropertyDescriptor 
KINESIS_FIREHOSE_DELIVERY_STREAM_NAME = new PropertyDescriptor.Builder()
+.name("Amazon Kinesis Firehose Delivery Stream Name")
+.description("The name of kinesis firehose delivery stream")
+.expressionLanguageSupported(false)
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor MAX_BUFFER_INTERVAL = new 
PropertyDescriptor.Builder()
+.name("Max Buffer Interval")
+.description("Buffering interval for messages (between 60 and 
900 seconds).")
+.defaultValue("60")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.sensitive(false)
+.build();
+
+public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
+.name("Max Buffer Size (MB)")
+.description("Buffering size for messages (between 1MB and 
128MB).")
+.defaultValue("128")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.sensitive(false)
+.build();
+
+public static final PropertyDescriptor BATCH_SIZE = new 
PropertyDescriptor.Builder()
--- End diff --

@apiri - I've corrected this based on your feedback.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread mans2singh

Github user mans2singh commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53893190
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/firehose/AbstractKinesisFirehoseProcessor.java
 ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor;
+
+import com.amazonaws.ClientConfiguration;
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.services.kinesisfirehose.AmazonKinesisFirehoseClient;
+
+/**
+ * This class provides processor the base class for kinesis firehose
+ */
+public abstract class AbstractKinesisFirehoseProcessor extends 
AbstractAWSCredentialsProviderProcessor {
+
+public static final PropertyDescriptor 
KINESIS_FIREHOSE_DELIVERY_STREAM_NAME = new PropertyDescriptor.Builder()
+.name("Amazon Kinesis Firehose Delivery Stream Name")
+.description("The name of kinesis firehose delivery stream")
+.expressionLanguageSupported(false)
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor MAX_BUFFER_INTERVAL = new 
PropertyDescriptor.Builder()
+.name("Max Buffer Interval")
+.description("Buffering interval for messages (between 60 and 
900 seconds).")
+.defaultValue("60")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
--- End diff --

@apiri - The max buffer size and interval are configuration properties and 
as you pointed out correctly not used in client.  I've removed them and also 
have updated validators to use createLongValidator


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[VOTE] Release Apache NiFi 0.5.1 (RC2)

2016-02-23 Thread Tony Kurc

Hello,
I am pleased to be calling this vote for the source release of Apache NiFi
nifi-0.5.1.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1076

The Git tag is nifi-0.5.1-RC2
The Git commit ID is 672211b87b4f1e52f8ee5153c26a467b555a331e
https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=672211b87b4f1e52f8ee5153c26a467b555a331e

This release candidate is a branch off of support/nifi-0.5.x at
e2005fa059fbe128e2e278cda5ed7a27ab6e1ec3

Checksums of nifi-0.5.1-source-release.zip:
MD5: 9139aaae5d0a42a0fbbb624c2e739cdd
SHA1: 374a24354f54c7b6c04ba0897f8e2e0b9a6a5127

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/tkurc.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

12 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12334887

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.5.1

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build from source, and test. Then
please vote:

[ ] +1 Release this package as nifi-0.5.1
[ ] +0 no opinion
[ ] -1 Do not release this package because because...

Thanks!
Tony

[GitHub] nifi pull request: NiFi-1481 Enhancement[ nifi.sh env]

2016-02-23 Thread PuspenduBanerjee

Github user PuspenduBanerjee commented on the pull request:

https://github.com/apache/nifi/pull/218#issuecomment-187999652
  
@trkurc Thanks. I shall keep it as is & look forward to see it as part of 
0.6.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] Release Apache NiFi 0.5.1 (RC1)

2016-02-23 Thread Tony Kurc

While I start working on putting together RC2, please consider checking out
e2005fa059fbe128e2e278cda5ed7a27ab6e1ec3, which is where I intend to branch
from.

On Tue, Feb 23, 2016 at 4:37 PM, Joe Witt  wrote:

> On the plus side the patch is running on a cluster and working really
> well.  The issue was really obvious once we dug in.
>
> https://issues.apache.org/jira/browse/NIFI-1557
>
> Thanks
> Joe
>
> On Tue, Feb 23, 2016 at 4:36 PM, Tony Kurc  wrote:
> > I believe that is enough of the pmc to cancel this vote, so I will do so
> > On Feb 23, 2016 4:28 PM, "Mark Payne"  wrote:
> >
> >> Agreed - I would recommend holding off the 0.5.1 release for this - it
> can
> >> be very
> >> frustrating when this occurs. Will change my vote also to -1.
> >>
> >>
> >> > On Feb 23, 2016, at 4:11 PM, Joe Witt  wrote:
> >> >
> >> > -1 (binding).
> >> >
> >> > I continued testing to include running on a cluster w/two-way SSL and
> >> > username/password via LDAP.  What I observed has been filed as this
> >> > JIRA https://issues.apache.org/jira/browse/NIFI-1557
> >> >
> >> > The root of the issue has been identified and am now testing against a
> >> > patch that has proven to address the problem.  Continuing to evaluate
> >> > behavior around this to ensure it is fully addressed.
> >> >
> >> > This issue is believed to have been present ever since we added
> >> > controller services and reporting tasks through the UI (version 0.1.0)
> >> > and has been causing confusion for folks for some time and can create
> >> > difficultly when restarting nodes or upgrading.  In my opinion this
> >> > warrants canceling 0.5.1 RC1 and instead addressing this issue and
> >> > producing an 0.5.1 RC2.
> >> >
> >> > Thanks
> >> > Joe
> >> >
> >> > On Tue, Feb 23, 2016 at 1:35 PM, Mark Payne 
> >> wrote:
> >> >> +1 (binding) Release this package as nifi-0.5.1
> >> >>
> >> >> Verified hashes & build w/ unit tests & contrib work. Verified that
> the
> >> initial issue that
> >> >> caused the 0.5.1 release (NIFI-1527) was addressed.
> >> >>
> >> >> -Mark
> >> >>
> >> >>
> >> >>> On Feb 23, 2016, at 12:35 PM, Joe Witt  wrote:
> >> >>>
> >> >>> +1 (binding) Release this package as nifi-0.5.1
> >> >>>
> >> >>> Ran through all steps outlined for verification and validated
> numerous
> >> >>> JIRAs on the release and confirmed the finding Sean had for the
> >> >>> sources/javadocs has been resolved.
> >> >>>
> >> >>> Thanks!
> >> >>>
> >> >>> On Tue, Feb 23, 2016 at 7:27 AM, Brian Ghigiarelli <
> >> briang...@gmail.com> wrote:
> >>  Upgrading to JDK 8 u74 works like a charm - everything builds and
> >> tests to
> >>  success. The fan is a lot quieter now too. Thanks for the pointer
> on
> >> the
> >>  JDK version!
> >> 
> >>  On Tue, Feb 23, 2016 at 7:08 AM Brian Ghigiarelli <
> >> briang...@gmail.com>
> >>  wrote:
> >> 
> >> > Thanks, Tony!
> >> >
> >> > Oracle JDK 8 u45 (1.8.0_45)
> >> >
> >> > At this point, I've tried the following MAVEN_OPTS for "mvn -T
> C2.0
> >> clean
> >> > install" and "mvn -T C4.0 clean install"
> >> >
> >> > -Xms1024m -Xmx3076m
> >> > -Xms128m -Xmx4096m
> >> > -Xmx1024m -Xmx4096m -XX:+UseG1GC -XX:+UseStringDeduplication
> >> > -Xmx4096m -XX:-UseGCOverheadLimit -XX:+UseG1GC
> >> > -Xmx3g -XX:-UseGCOverheadLimit
> >> >
> >> > All eventually fail in the same spot:
> >> >
> >> > Tests run: 139, Failures: 1, Errors: 0, Skipped: 1, Time elapsed:
> >> 154.539
> >> > sec <<< FAILURE! - in TestSuite
> >> > testManyFilesOpened on
> >> >
> >>
> testManyFilesOpened(org.apache.nifi.controller.repository.TestStandardProcessSession)(org.apache.nifi.controller.repository.TestStandardProcessSession)
> >> > Time elapsed: 59.672 sec  <<< FAILURE!
> >> > java.lang.OutOfMemoryError: GC overhead limit exceeded
> >> > at
> >> >
> >>
> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:61)
> >> > at
> >> >
> >>
> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:46)
> >> > at
> >> >
> >>
> org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2009)
> >> > at
> >> >
> >>
> org.apache.nifi.controller.repository.TestStandardProcessSession.testManyFilesOpened(TestStandardProcessSession.java:775)
> >> >
> >> > This is running on a MacBook Pro OS X 10.11.3 (El Capitan) with a
> >> 3GHz i7
> >> > processor and 16GB memory.
> >> >
> >> > On Tue, Feb 23, 2016 at 6:35 AM Tony Kurc 
> wrote:
> >> >
> >> >> often that error is because the heap is overcommitted What
> >> version of
> >> >> the JDK are you attempting to build with, and what MAVEN_OPTS are
> >> you
> >> >> using? I built successfully with MAVEN_OPTS="-Xms1024m
> -Xmx3076m".
> >> >>
> >> >> On Tue, Feb 23, 2016 at 6:04 AM, Brian Ghigiarelli <
> >> briang...@gmail.com>
> >> >> wrote:
> >> >>
> >> >>> Hi P

Re: Logging issue in Jira

2016-02-23 Thread Devin Fisher

Thanks, that got me what I needed. Not super familiar with Jira. The Issue
has been logged (NIFI-1560 
).
Devin

On Mon, Feb 22, 2016 at 1:08 PM, Matt Gilman 
wrote:

> Devin,
>
> Thanks for reporting and showing interest in NiFi!
>
> There should be a button in the toolbar at the top that says Create. I
> believe this button is only visible if you are logged in. Have you created
> a JIRA account?
>
> Matt
>
> On Mon, Feb 22, 2016 at 3:02 PM, Devin Fisher <
> devin.fis...@perfectsearchcorp.com> wrote:
>
> > I noticed a small copy and past error in LdapProvider class. (the error
> > message is not using the right variable) I tried to create a ticket in
> Jira
> > but don't see how to do so. Look through the Contributor Guide and did
> not
> > see info on how to log a bug. What is the best approach to log an issue?
> >
> > Devin
> >
>

[CANCEL][VOTE] Release Apache NiFi 0.5.1 (RC1)

2016-02-23 Thread Tony Kurc

Due to issues documented here [1] uncovered during RC1 testing, the vote is
cancelled

Thank you for those assisting is getting this release together, we're close!

1. https://issues.apache.org/jira/browse/NIFI-1557

Re: [VOTE] Release Apache NiFi 0.5.1 (RC1)

2016-02-23 Thread Joe Witt

On the plus side the patch is running on a cluster and working really
well.  The issue was really obvious once we dug in.

https://issues.apache.org/jira/browse/NIFI-1557

Thanks
Joe

On Tue, Feb 23, 2016 at 4:36 PM, Tony Kurc  wrote:
> I believe that is enough of the pmc to cancel this vote, so I will do so
> On Feb 23, 2016 4:28 PM, "Mark Payne"  wrote:
>
>> Agreed - I would recommend holding off the 0.5.1 release for this - it can
>> be very
>> frustrating when this occurs. Will change my vote also to -1.
>>
>>
>> > On Feb 23, 2016, at 4:11 PM, Joe Witt  wrote:
>> >
>> > -1 (binding).
>> >
>> > I continued testing to include running on a cluster w/two-way SSL and
>> > username/password via LDAP.  What I observed has been filed as this
>> > JIRA https://issues.apache.org/jira/browse/NIFI-1557
>> >
>> > The root of the issue has been identified and am now testing against a
>> > patch that has proven to address the problem.  Continuing to evaluate
>> > behavior around this to ensure it is fully addressed.
>> >
>> > This issue is believed to have been present ever since we added
>> > controller services and reporting tasks through the UI (version 0.1.0)
>> > and has been causing confusion for folks for some time and can create
>> > difficultly when restarting nodes or upgrading.  In my opinion this
>> > warrants canceling 0.5.1 RC1 and instead addressing this issue and
>> > producing an 0.5.1 RC2.
>> >
>> > Thanks
>> > Joe
>> >
>> > On Tue, Feb 23, 2016 at 1:35 PM, Mark Payne 
>> wrote:
>> >> +1 (binding) Release this package as nifi-0.5.1
>> >>
>> >> Verified hashes & build w/ unit tests & contrib work. Verified that the
>> initial issue that
>> >> caused the 0.5.1 release (NIFI-1527) was addressed.
>> >>
>> >> -Mark
>> >>
>> >>
>> >>> On Feb 23, 2016, at 12:35 PM, Joe Witt  wrote:
>> >>>
>> >>> +1 (binding) Release this package as nifi-0.5.1
>> >>>
>> >>> Ran through all steps outlined for verification and validated numerous
>> >>> JIRAs on the release and confirmed the finding Sean had for the
>> >>> sources/javadocs has been resolved.
>> >>>
>> >>> Thanks!
>> >>>
>> >>> On Tue, Feb 23, 2016 at 7:27 AM, Brian Ghigiarelli <
>> briang...@gmail.com> wrote:
>>  Upgrading to JDK 8 u74 works like a charm - everything builds and
>> tests to
>>  success. The fan is a lot quieter now too. Thanks for the pointer on
>> the
>>  JDK version!
>> 
>>  On Tue, Feb 23, 2016 at 7:08 AM Brian Ghigiarelli <
>> briang...@gmail.com>
>>  wrote:
>> 
>> > Thanks, Tony!
>> >
>> > Oracle JDK 8 u45 (1.8.0_45)
>> >
>> > At this point, I've tried the following MAVEN_OPTS for "mvn -T C2.0
>> clean
>> > install" and "mvn -T C4.0 clean install"
>> >
>> > -Xms1024m -Xmx3076m
>> > -Xms128m -Xmx4096m
>> > -Xmx1024m -Xmx4096m -XX:+UseG1GC -XX:+UseStringDeduplication
>> > -Xmx4096m -XX:-UseGCOverheadLimit -XX:+UseG1GC
>> > -Xmx3g -XX:-UseGCOverheadLimit
>> >
>> > All eventually fail in the same spot:
>> >
>> > Tests run: 139, Failures: 1, Errors: 0, Skipped: 1, Time elapsed:
>> 154.539
>> > sec <<< FAILURE! - in TestSuite
>> > testManyFilesOpened on
>> >
>> testManyFilesOpened(org.apache.nifi.controller.repository.TestStandardProcessSession)(org.apache.nifi.controller.repository.TestStandardProcessSession)
>> > Time elapsed: 59.672 sec  <<< FAILURE!
>> > java.lang.OutOfMemoryError: GC overhead limit exceeded
>> > at
>> >
>> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:61)
>> > at
>> >
>> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:46)
>> > at
>> >
>> org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2009)
>> > at
>> >
>> org.apache.nifi.controller.repository.TestStandardProcessSession.testManyFilesOpened(TestStandardProcessSession.java:775)
>> >
>> > This is running on a MacBook Pro OS X 10.11.3 (El Capitan) with a
>> 3GHz i7
>> > processor and 16GB memory.
>> >
>> > On Tue, Feb 23, 2016 at 6:35 AM Tony Kurc  wrote:
>> >
>> >> often that error is because the heap is overcommitted What
>> version of
>> >> the JDK are you attempting to build with, and what MAVEN_OPTS are
>> you
>> >> using? I built successfully with MAVEN_OPTS="-Xms1024m -Xmx3076m".
>> >>
>> >> On Tue, Feb 23, 2016 at 6:04 AM, Brian Ghigiarelli <
>> briang...@gmail.com>
>> >> wrote:
>> >>
>> >>> Hi Pierre,
>> >>>
>> >>> I'm hitting the same issues on Mac with the master branch
>> (specifically
>> >>> while it runs testManyOpenedFiles in nifi-framework-core). Been
>> tuning
>> >> Xmx,
>> >>> ulimits, and G1GC to no avail. Perhaps we could continue that
>> >> investigation
>> >>> on a separate non-voting thread.
>> >>>
>> >>> Thanks,
>> >>> Brian
>> >>>
>> >>> On Tue, Feb 23, 2016 at 5:53 AM Pierre Villard <
>> >>> pierre.vi

Re: [VOTE] Release Apache NiFi 0.5.1 (RC1)

2016-02-23 Thread Tony Kurc

I believe that is enough of the pmc to cancel this vote, so I will do so
On Feb 23, 2016 4:28 PM, "Mark Payne"  wrote:

> Agreed - I would recommend holding off the 0.5.1 release for this - it can
> be very
> frustrating when this occurs. Will change my vote also to -1.
>
>
> > On Feb 23, 2016, at 4:11 PM, Joe Witt  wrote:
> >
> > -1 (binding).
> >
> > I continued testing to include running on a cluster w/two-way SSL and
> > username/password via LDAP.  What I observed has been filed as this
> > JIRA https://issues.apache.org/jira/browse/NIFI-1557
> >
> > The root of the issue has been identified and am now testing against a
> > patch that has proven to address the problem.  Continuing to evaluate
> > behavior around this to ensure it is fully addressed.
> >
> > This issue is believed to have been present ever since we added
> > controller services and reporting tasks through the UI (version 0.1.0)
> > and has been causing confusion for folks for some time and can create
> > difficultly when restarting nodes or upgrading.  In my opinion this
> > warrants canceling 0.5.1 RC1 and instead addressing this issue and
> > producing an 0.5.1 RC2.
> >
> > Thanks
> > Joe
> >
> > On Tue, Feb 23, 2016 at 1:35 PM, Mark Payne 
> wrote:
> >> +1 (binding) Release this package as nifi-0.5.1
> >>
> >> Verified hashes & build w/ unit tests & contrib work. Verified that the
> initial issue that
> >> caused the 0.5.1 release (NIFI-1527) was addressed.
> >>
> >> -Mark
> >>
> >>
> >>> On Feb 23, 2016, at 12:35 PM, Joe Witt  wrote:
> >>>
> >>> +1 (binding) Release this package as nifi-0.5.1
> >>>
> >>> Ran through all steps outlined for verification and validated numerous
> >>> JIRAs on the release and confirmed the finding Sean had for the
> >>> sources/javadocs has been resolved.
> >>>
> >>> Thanks!
> >>>
> >>> On Tue, Feb 23, 2016 at 7:27 AM, Brian Ghigiarelli <
> briang...@gmail.com> wrote:
>  Upgrading to JDK 8 u74 works like a charm - everything builds and
> tests to
>  success. The fan is a lot quieter now too. Thanks for the pointer on
> the
>  JDK version!
> 
>  On Tue, Feb 23, 2016 at 7:08 AM Brian Ghigiarelli <
> briang...@gmail.com>
>  wrote:
> 
> > Thanks, Tony!
> >
> > Oracle JDK 8 u45 (1.8.0_45)
> >
> > At this point, I've tried the following MAVEN_OPTS for "mvn -T C2.0
> clean
> > install" and "mvn -T C4.0 clean install"
> >
> > -Xms1024m -Xmx3076m
> > -Xms128m -Xmx4096m
> > -Xmx1024m -Xmx4096m -XX:+UseG1GC -XX:+UseStringDeduplication
> > -Xmx4096m -XX:-UseGCOverheadLimit -XX:+UseG1GC
> > -Xmx3g -XX:-UseGCOverheadLimit
> >
> > All eventually fail in the same spot:
> >
> > Tests run: 139, Failures: 1, Errors: 0, Skipped: 1, Time elapsed:
> 154.539
> > sec <<< FAILURE! - in TestSuite
> > testManyFilesOpened on
> >
> testManyFilesOpened(org.apache.nifi.controller.repository.TestStandardProcessSession)(org.apache.nifi.controller.repository.TestStandardProcessSession)
> > Time elapsed: 59.672 sec  <<< FAILURE!
> > java.lang.OutOfMemoryError: GC overhead limit exceeded
> > at
> >
> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:61)
> > at
> >
> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:46)
> > at
> >
> org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2009)
> > at
> >
> org.apache.nifi.controller.repository.TestStandardProcessSession.testManyFilesOpened(TestStandardProcessSession.java:775)
> >
> > This is running on a MacBook Pro OS X 10.11.3 (El Capitan) with a
> 3GHz i7
> > processor and 16GB memory.
> >
> > On Tue, Feb 23, 2016 at 6:35 AM Tony Kurc  wrote:
> >
> >> often that error is because the heap is overcommitted What
> version of
> >> the JDK are you attempting to build with, and what MAVEN_OPTS are
> you
> >> using? I built successfully with MAVEN_OPTS="-Xms1024m -Xmx3076m".
> >>
> >> On Tue, Feb 23, 2016 at 6:04 AM, Brian Ghigiarelli <
> briang...@gmail.com>
> >> wrote:
> >>
> >>> Hi Pierre,
> >>>
> >>> I'm hitting the same issues on Mac with the master branch
> (specifically
> >>> while it runs testManyOpenedFiles in nifi-framework-core). Been
> tuning
> >> Xmx,
> >>> ulimits, and G1GC to no avail. Perhaps we could continue that
> >> investigation
> >>> on a separate non-voting thread.
> >>>
> >>> Thanks,
> >>> Brian
> >>>
> >>> On Tue, Feb 23, 2016 at 5:53 AM Pierre Villard <
> >>> pierre.villard...@gmail.com>
> >>> wrote:
> >>>
>  Hi all,
> 
>  I got some issues when building the version (on Windows):
>  - with 'mvn clean install -Pcontrib-check', I got GC overhead
> limit
> >> error
>  on unit tests in framework-core (I didn't manage to get it
> working by
>  changing options in Maven but I g

Re: [VOTE] Release Apache NiFi 0.5.1 (RC1)

2016-02-23 Thread Mark Payne

Agreed - I would recommend holding off the 0.5.1 release for this - it can be 
very
frustrating when this occurs. Will change my vote also to -1.


> On Feb 23, 2016, at 4:11 PM, Joe Witt  wrote:
> 
> -1 (binding).
> 
> I continued testing to include running on a cluster w/two-way SSL and
> username/password via LDAP.  What I observed has been filed as this
> JIRA https://issues.apache.org/jira/browse/NIFI-1557
> 
> The root of the issue has been identified and am now testing against a
> patch that has proven to address the problem.  Continuing to evaluate
> behavior around this to ensure it is fully addressed.
> 
> This issue is believed to have been present ever since we added
> controller services and reporting tasks through the UI (version 0.1.0)
> and has been causing confusion for folks for some time and can create
> difficultly when restarting nodes or upgrading.  In my opinion this
> warrants canceling 0.5.1 RC1 and instead addressing this issue and
> producing an 0.5.1 RC2.
> 
> Thanks
> Joe
> 
> On Tue, Feb 23, 2016 at 1:35 PM, Mark Payne  wrote:
>> +1 (binding) Release this package as nifi-0.5.1
>> 
>> Verified hashes & build w/ unit tests & contrib work. Verified that the 
>> initial issue that
>> caused the 0.5.1 release (NIFI-1527) was addressed.
>> 
>> -Mark
>> 
>> 
>>> On Feb 23, 2016, at 12:35 PM, Joe Witt  wrote:
>>> 
>>> +1 (binding) Release this package as nifi-0.5.1
>>> 
>>> Ran through all steps outlined for verification and validated numerous
>>> JIRAs on the release and confirmed the finding Sean had for the
>>> sources/javadocs has been resolved.
>>> 
>>> Thanks!
>>> 
>>> On Tue, Feb 23, 2016 at 7:27 AM, Brian Ghigiarelli  
>>> wrote:
 Upgrading to JDK 8 u74 works like a charm - everything builds and tests to
 success. The fan is a lot quieter now too. Thanks for the pointer on the
 JDK version!
 
 On Tue, Feb 23, 2016 at 7:08 AM Brian Ghigiarelli 
 wrote:
 
> Thanks, Tony!
> 
> Oracle JDK 8 u45 (1.8.0_45)
> 
> At this point, I've tried the following MAVEN_OPTS for "mvn -T C2.0 clean
> install" and "mvn -T C4.0 clean install"
> 
> -Xms1024m -Xmx3076m
> -Xms128m -Xmx4096m
> -Xmx1024m -Xmx4096m -XX:+UseG1GC -XX:+UseStringDeduplication
> -Xmx4096m -XX:-UseGCOverheadLimit -XX:+UseG1GC
> -Xmx3g -XX:-UseGCOverheadLimit
> 
> All eventually fail in the same spot:
> 
> Tests run: 139, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 154.539
> sec <<< FAILURE! - in TestSuite
> testManyFilesOpened on
> testManyFilesOpened(org.apache.nifi.controller.repository.TestStandardProcessSession)(org.apache.nifi.controller.repository.TestStandardProcessSession)
> Time elapsed: 59.672 sec  <<< FAILURE!
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at
> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:61)
> at
> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:46)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2009)
> at
> org.apache.nifi.controller.repository.TestStandardProcessSession.testManyFilesOpened(TestStandardProcessSession.java:775)
> 
> This is running on a MacBook Pro OS X 10.11.3 (El Capitan) with a 3GHz i7
> processor and 16GB memory.
> 
> On Tue, Feb 23, 2016 at 6:35 AM Tony Kurc  wrote:
> 
>> often that error is because the heap is overcommitted What version of
>> the JDK are you attempting to build with, and what MAVEN_OPTS are you
>> using? I built successfully with MAVEN_OPTS="-Xms1024m -Xmx3076m".
>> 
>> On Tue, Feb 23, 2016 at 6:04 AM, Brian Ghigiarelli 
>> wrote:
>> 
>>> Hi Pierre,
>>> 
>>> I'm hitting the same issues on Mac with the master branch (specifically
>>> while it runs testManyOpenedFiles in nifi-framework-core). Been tuning
>> Xmx,
>>> ulimits, and G1GC to no avail. Perhaps we could continue that
>> investigation
>>> on a separate non-voting thread.
>>> 
>>> Thanks,
>>> Brian
>>> 
>>> On Tue, Feb 23, 2016 at 5:53 AM Pierre Villard <
>>> pierre.villard...@gmail.com>
>>> wrote:
>>> 
 Hi all,
 
 I got some issues when building the version (on Windows):
 - with 'mvn clean install -Pcontrib-check', I got GC overhead limit
>> error
 on unit tests in framework-core (I didn't manage to get it working by
 changing options in Maven but I guess I missed something with my own
>>> local
 environment)
 - with 'mvn clean install -Pcontrib-check -DskipTests', I got the
>>> following
 error : [ERROR] Failed to execute goal
 org.apache.rat:apache-rat-plugin:0.11:check (default) on project
 nifi-standard-processors: Too many files with unapproved license: 1
>> See
>>> RAT
 report (It comes from
>

Re: [VOTE] Release Apache NiFi 0.5.1 (RC1)

2016-02-23 Thread Joe Witt

-1 (binding).

I continued testing to include running on a cluster w/two-way SSL and
username/password via LDAP.  What I observed has been filed as this
JIRA https://issues.apache.org/jira/browse/NIFI-1557

The root of the issue has been identified and am now testing against a
patch that has proven to address the problem.  Continuing to evaluate
behavior around this to ensure it is fully addressed.

This issue is believed to have been present ever since we added
controller services and reporting tasks through the UI (version 0.1.0)
and has been causing confusion for folks for some time and can create
difficultly when restarting nodes or upgrading.  In my opinion this
warrants canceling 0.5.1 RC1 and instead addressing this issue and
producing an 0.5.1 RC2.

Thanks
Joe

On Tue, Feb 23, 2016 at 1:35 PM, Mark Payne  wrote:
> +1 (binding) Release this package as nifi-0.5.1
>
> Verified hashes & build w/ unit tests & contrib work. Verified that the 
> initial issue that
> caused the 0.5.1 release (NIFI-1527) was addressed.
>
> -Mark
>
>
>> On Feb 23, 2016, at 12:35 PM, Joe Witt  wrote:
>>
>> +1 (binding) Release this package as nifi-0.5.1
>>
>> Ran through all steps outlined for verification and validated numerous
>> JIRAs on the release and confirmed the finding Sean had for the
>> sources/javadocs has been resolved.
>>
>> Thanks!
>>
>> On Tue, Feb 23, 2016 at 7:27 AM, Brian Ghigiarelli  
>> wrote:
>>> Upgrading to JDK 8 u74 works like a charm - everything builds and tests to
>>> success. The fan is a lot quieter now too. Thanks for the pointer on the
>>> JDK version!
>>>
>>> On Tue, Feb 23, 2016 at 7:08 AM Brian Ghigiarelli 
>>> wrote:
>>>
 Thanks, Tony!

 Oracle JDK 8 u45 (1.8.0_45)

 At this point, I've tried the following MAVEN_OPTS for "mvn -T C2.0 clean
 install" and "mvn -T C4.0 clean install"

 -Xms1024m -Xmx3076m
 -Xms128m -Xmx4096m
 -Xmx1024m -Xmx4096m -XX:+UseG1GC -XX:+UseStringDeduplication
 -Xmx4096m -XX:-UseGCOverheadLimit -XX:+UseG1GC
 -Xmx3g -XX:-UseGCOverheadLimit

 All eventually fail in the same spot:

 Tests run: 139, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 154.539
 sec <<< FAILURE! - in TestSuite
 testManyFilesOpened on
 testManyFilesOpened(org.apache.nifi.controller.repository.TestStandardProcessSession)(org.apache.nifi.controller.repository.TestStandardProcessSession)
 Time elapsed: 59.672 sec  <<< FAILURE!
 java.lang.OutOfMemoryError: GC overhead limit exceeded
 at
 org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:61)
 at
 org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:46)
 at
 org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2009)
 at
 org.apache.nifi.controller.repository.TestStandardProcessSession.testManyFilesOpened(TestStandardProcessSession.java:775)

 This is running on a MacBook Pro OS X 10.11.3 (El Capitan) with a 3GHz i7
 processor and 16GB memory.

 On Tue, Feb 23, 2016 at 6:35 AM Tony Kurc  wrote:

> often that error is because the heap is overcommitted What version of
> the JDK are you attempting to build with, and what MAVEN_OPTS are you
> using? I built successfully with MAVEN_OPTS="-Xms1024m -Xmx3076m".
>
> On Tue, Feb 23, 2016 at 6:04 AM, Brian Ghigiarelli 
> wrote:
>
>> Hi Pierre,
>>
>> I'm hitting the same issues on Mac with the master branch (specifically
>> while it runs testManyOpenedFiles in nifi-framework-core). Been tuning
> Xmx,
>> ulimits, and G1GC to no avail. Perhaps we could continue that
> investigation
>> on a separate non-voting thread.
>>
>> Thanks,
>> Brian
>>
>> On Tue, Feb 23, 2016 at 5:53 AM Pierre Villard <
>> pierre.villard...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I got some issues when building the version (on Windows):
>>> - with 'mvn clean install -Pcontrib-check', I got GC overhead limit
> error
>>> on unit tests in framework-core (I didn't manage to get it working by
>>> changing options in Maven but I guess I missed something with my own
>> local
>>> environment)
>>> - with 'mvn clean install -Pcontrib-check -DskipTests', I got the
>> following
>>> error : [ERROR] Failed to execute goal
>>> org.apache.rat:apache-rat-plugin:0.11:check (default) on project
>>> nifi-standard-processors: Too many files with unapproved license: 1
> See
>> RAT
>>> report (It comes from
>>>
>>>
>>
> nifi-0.5.1/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestEncryptContent/unsalted_128_raw.enc)
>>> - with 'mvn clean install -DskipTests', it is OK.
>>>
>>> Otherwise, as far as I looked, it seems OK to me.
>>>
>>>
>>>
>>>
>>> 2016-02-23 5:40 GMT+01:00 Tony Kurc :
>>>
>

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread apiri

Github user apiri commented on the pull request:

https://github.com/apache/nifi/pull/213#issuecomment-187907196
  
@mans2singh Did an initial look over the code.  The approach seems good but 
would like to understand more about the batching process and how we can perform 
this in a safe manner that does not exhaust heap so readily.  

Also, would you please be able to edit the PR name to something with 
NIFI-1495 in it so that hopefully the JIRA integration will link and include 
those items on the issue.  It is unneeded to keep the reference to NIFI-1489 as 
those changes have been merged and PR closed in #209 

Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread apiri

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53850051
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/firehose/ITPutKinesisFirehose.java
 ---
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import java.util.List;
+
+import org.apache.nifi.processors.aws.s3.FetchS3Object;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * This test contains both unit and integration test (integration tests 
are ignored by default)
+ */
+public class ITPutKinesisFirehose {
+
+private TestRunner runner;
+protected final static String CREDENTIALS_FILE = 
System.getProperty("user.home") + "/aws-credentials.properties";
+
+@Before
+public void setUp() throws Exception {
+runner = TestRunners.newTestRunner(PutKinesisFirehose.class);
+runner.setProperty(PutKinesisFirehose.ACCESS_KEY, "abcd");
+runner.setProperty(PutKinesisFirehose.SECRET_KEY, "secret key");
+
runner.setProperty(PutKinesisFirehose.KINESIS_FIREHOSE_DELIVERY_STREAM_NAME, 
"deliveryName");
+runner.assertValid();
+}
+
+@After
+public void tearDown() throws Exception {
+runner = null;
+}
+
+@Test
+public void testCustomValidateBatchSize1Valid() {
+runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "1");
+runner.assertValid();
+}
+
+@Test
+public void testCustomValidateBatchSize500Valid() {
+runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "500");
+runner.assertValid();
+}
+@Test
+public void testCustomValidateBatchSize501InValid() {
+runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "501");
+runner.assertNotValid();
+}
+
+@Test
+public void testCustomValidateBufferSize1Valid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_SIZE, "1");
+runner.assertValid();
+}
+
+@Test
+public void testCustomValidateBufferSize128Valid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_SIZE, "128");
+runner.assertValid();
+}
+@Test
+public void testCustomValidateBufferSize129InValid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_SIZE, "129");
+runner.assertNotValid();
+}
+
+@Test
+public void testCustomValidateBufferInterval900Valid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_INTERVAL, "900");
+runner.assertValid();
+}
+
+@Test
+public void testCustomValidateBufferInterval60Valid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_INTERVAL, "60");
+runner.assertValid();
+}
+
+@Test
+public void testCustomValidateBufferInterval901InValid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_INTERVAL, "901");
+runner.assertNotValid();
+}
+
+@Test
+public void testCustomValidateBufferInterval59InValid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_INTERVAL, "59");
+runner.assertNotValid();
+}
+
+/**
+ * Comment out ignore for integration tests (requires creds files)
+ */
+@Test
+@Ignore
+public void testIntegrationSuccess() throws Exception {
+runner = TestRunners.newTestRunner(PutKinesisFirehose.class);
+runner.setProperty(PutKinesisFirehose.CREDENTIALS_FILE, 
CREDENTIALS_FILE);
+
runner.setProperty(PutKinesisFirehose.KINESIS_FIR

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread apiri

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53849876
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/firehose/ITPutKinesisFirehose.java
 ---
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import java.util.List;
+
+import org.apache.nifi.processors.aws.s3.FetchS3Object;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * This test contains both unit and integration test (integration tests 
are ignored by default)
+ */
+public class ITPutKinesisFirehose {
--- End diff --

customValidate tests and similar, if still needed or desired, should go 
into a standard TestPutKinesisFirehose class so that they are run on each build


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread apiri

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53849994
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/firehose/ITPutKinesisFirehose.java
 ---
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import java.util.List;
+
+import org.apache.nifi.processors.aws.s3.FetchS3Object;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * This test contains both unit and integration test (integration tests 
are ignored by default)
+ */
+public class ITPutKinesisFirehose {
+
+private TestRunner runner;
+protected final static String CREDENTIALS_FILE = 
System.getProperty("user.home") + "/aws-credentials.properties";
+
+@Before
+public void setUp() throws Exception {
+runner = TestRunners.newTestRunner(PutKinesisFirehose.class);
+runner.setProperty(PutKinesisFirehose.ACCESS_KEY, "abcd");
+runner.setProperty(PutKinesisFirehose.SECRET_KEY, "secret key");
+
runner.setProperty(PutKinesisFirehose.KINESIS_FIREHOSE_DELIVERY_STREAM_NAME, 
"deliveryName");
+runner.assertValid();
+}
+
+@After
+public void tearDown() throws Exception {
+runner = null;
+}
+
+@Test
+public void testCustomValidateBatchSize1Valid() {
+runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "1");
+runner.assertValid();
+}
+
+@Test
+public void testCustomValidateBatchSize500Valid() {
+runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "500");
+runner.assertValid();
+}
+@Test
+public void testCustomValidateBatchSize501InValid() {
+runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "501");
+runner.assertNotValid();
+}
+
+@Test
+public void testCustomValidateBufferSize1Valid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_SIZE, "1");
+runner.assertValid();
+}
+
+@Test
+public void testCustomValidateBufferSize128Valid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_SIZE, "128");
+runner.assertValid();
+}
+@Test
+public void testCustomValidateBufferSize129InValid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_SIZE, "129");
+runner.assertNotValid();
+}
+
+@Test
+public void testCustomValidateBufferInterval900Valid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_INTERVAL, "900");
+runner.assertValid();
+}
+
+@Test
+public void testCustomValidateBufferInterval60Valid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_INTERVAL, "60");
+runner.assertValid();
+}
+
+@Test
+public void testCustomValidateBufferInterval901InValid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_INTERVAL, "901");
+runner.assertNotValid();
+}
+
+@Test
+public void testCustomValidateBufferInterval59InValid() {
+runner.setProperty(PutKinesisFirehose.MAX_BUFFER_INTERVAL, "59");
+runner.assertNotValid();
+}
+
+/**
+ * Comment out ignore for integration tests (requires creds files)
+ */
+@Test
+@Ignore
--- End diff --

The profile to run ITs is not enabled by default and this Ignore is 
unneeded.  Would prefer to have removed so that no code changes are needed to 
run them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread apiri

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53849365
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/firehose/PutKinesisFirehose.java
 ---
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import com.amazonaws.services.kinesisfirehose.AmazonKinesisFirehoseClient;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchRequest;
+import 
com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResponseEntry;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResult;
+import com.amazonaws.services.kinesisfirehose.model.Record;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"amazon", "aws", "firehose", "kinesis", "put", "stream"})
+@CapabilityDescription("Sends the contents to a specified Amazon Kinesis 
Firehose. "
++ "In order to send data to firehose, the firehose delivery stream 
name has to be specified.")
+@WritesAttributes({
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.message", 
description = "Error message on posting message to AWS Kinesis Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.code", 
description = "Error code for the message when posting to AWS Kinesis 
Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.record.id", 
description = "Record id of the message posted to Kinesis Firehose")})
+public class PutKinesisFirehose extends AbstractKinesisFirehoseProcessor {
+
+/**
+ * Kinesis put record response error message
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_MESSAGE = 
"aws.kinesis.firehose.error.message";
+
+/**
+ * Kinesis put record response error code
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_CODE = 
"aws.kinesis.firehose.error.code";
+
+/**
+ * Kinesis put record response record id
+ */
+public static final String AWS_KINESIS_FIREHOSE_RECORD_ID = 
"aws.kinesis.firehose.record.id";
+
+public static final List properties = 
Collections.unmodifiableList(
+Arrays.asList(KINESIS_FIREHOSE_DELIVERY_STREAM_NAME, 
MAX_BUFFER_INTERVAL,
+  MAX_BUFFER_SIZE, BATCH_SIZE, REGION, ACCESS_KEY, 
SECRET_KEY, CREDENTIALS_FILE, AWS_CREDENTIALS_PROVIDER_SERVICE, TIMEOUT,
+  PROXY_HOST,PROXY_HOST_PORT));
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@Override
+protected Collection customValidate(final 
ValidationContext validationContext) {
+final List problems = new 
ArrayList<>(super.customValidate(validationContext));

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread apiri

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53846413
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/firehose/ITPutKinesisFirehose.java
 ---
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import java.util.List;
+
+import org.apache.nifi.processors.aws.s3.FetchS3Object;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+
+/**
+ * This test contains both unit and integration test (integration tests 
are ignored by default)
+ */
+public class ITPutKinesisFirehose {
+
+private TestRunner runner;
+protected final static String CREDENTIALS_FILE = 
System.getProperty("user.home") + "/aws-credentials.properties";
+
+@Before
+public void setUp() throws Exception {
+runner = TestRunners.newTestRunner(PutKinesisFirehose.class);
+runner.setProperty(PutKinesisFirehose.ACCESS_KEY, "abcd");
+runner.setProperty(PutKinesisFirehose.SECRET_KEY, "secret key");
+
runner.setProperty(PutKinesisFirehose.KINESIS_FIREHOSE_DELIVERY_STREAM_NAME, 
"deliveryName");
+runner.assertValid();
+}
+
+@After
+public void tearDown() throws Exception {
+runner = null;
+}
+
+@Test
+public void testCustomValidateBatchSize1Valid() {
--- End diff --

These customValidate tests with the above improvements likely add little 
and could be removed with adjusted validators.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread apiri

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53845533
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/firehose/PutKinesisFirehose.java
 ---
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import com.amazonaws.services.kinesisfirehose.AmazonKinesisFirehoseClient;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchRequest;
+import 
com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResponseEntry;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResult;
+import com.amazonaws.services.kinesisfirehose.model.Record;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"amazon", "aws", "firehose", "kinesis", "put", "stream"})
+@CapabilityDescription("Sends the contents to a specified Amazon Kinesis 
Firehose. "
++ "In order to send data to firehose, the firehose delivery stream 
name has to be specified.")
+@WritesAttributes({
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.message", 
description = "Error message on posting message to AWS Kinesis Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.code", 
description = "Error code for the message when posting to AWS Kinesis 
Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.record.id", 
description = "Record id of the message posted to Kinesis Firehose")})
+public class PutKinesisFirehose extends AbstractKinesisFirehoseProcessor {
+
+/**
+ * Kinesis put record response error message
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_MESSAGE = 
"aws.kinesis.firehose.error.message";
+
+/**
+ * Kinesis put record response error code
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_CODE = 
"aws.kinesis.firehose.error.code";
+
+/**
+ * Kinesis put record response record id
+ */
+public static final String AWS_KINESIS_FIREHOSE_RECORD_ID = 
"aws.kinesis.firehose.record.id";
+
+public static final List properties = 
Collections.unmodifiableList(
+Arrays.asList(KINESIS_FIREHOSE_DELIVERY_STREAM_NAME, 
MAX_BUFFER_INTERVAL,
+  MAX_BUFFER_SIZE, BATCH_SIZE, REGION, ACCESS_KEY, 
SECRET_KEY, CREDENTIALS_FILE, AWS_CREDENTIALS_PROVIDER_SERVICE, TIMEOUT,
+  PROXY_HOST,PROXY_HOST_PORT));
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@Override
+protected Collection customValidate(final 
ValidationContext validationContext) {
+final List problems = new 
ArrayList<>(super.customValidate(validationContext));

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread apiri

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53845261
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/firehose/PutKinesisFirehose.java
 ---
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import com.amazonaws.services.kinesisfirehose.AmazonKinesisFirehoseClient;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchRequest;
+import 
com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResponseEntry;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResult;
+import com.amazonaws.services.kinesisfirehose.model.Record;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"amazon", "aws", "firehose", "kinesis", "put", "stream"})
+@CapabilityDescription("Sends the contents to a specified Amazon Kinesis 
Firehose. "
++ "In order to send data to firehose, the firehose delivery stream 
name has to be specified.")
+@WritesAttributes({
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.message", 
description = "Error message on posting message to AWS Kinesis Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.code", 
description = "Error code for the message when posting to AWS Kinesis 
Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.record.id", 
description = "Record id of the message posted to Kinesis Firehose")})
+public class PutKinesisFirehose extends AbstractKinesisFirehoseProcessor {
+
+/**
+ * Kinesis put record response error message
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_MESSAGE = 
"aws.kinesis.firehose.error.message";
+
+/**
+ * Kinesis put record response error code
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_CODE = 
"aws.kinesis.firehose.error.code";
+
+/**
+ * Kinesis put record response record id
+ */
+public static final String AWS_KINESIS_FIREHOSE_RECORD_ID = 
"aws.kinesis.firehose.record.id";
+
+public static final List properties = 
Collections.unmodifiableList(
+Arrays.asList(KINESIS_FIREHOSE_DELIVERY_STREAM_NAME, 
MAX_BUFFER_INTERVAL,
+  MAX_BUFFER_SIZE, BATCH_SIZE, REGION, ACCESS_KEY, 
SECRET_KEY, CREDENTIALS_FILE, AWS_CREDENTIALS_PROVIDER_SERVICE, TIMEOUT,
--- End diff --

If BUFFER_INTERVAL and BUFFER_SIZE are to go unused, they will also need to 
be removed from here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastr

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread apiri

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53845184
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/firehose/PutKinesisFirehose.java
 ---
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import com.amazonaws.services.kinesisfirehose.AmazonKinesisFirehoseClient;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchRequest;
+import 
com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResponseEntry;
+import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchResult;
+import com.amazonaws.services.kinesisfirehose.model.Record;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"amazon", "aws", "firehose", "kinesis", "put", "stream"})
+@CapabilityDescription("Sends the contents to a specified Amazon Kinesis 
Firehose. "
++ "In order to send data to firehose, the firehose delivery stream 
name has to be specified.")
+@WritesAttributes({
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.message", 
description = "Error message on posting message to AWS Kinesis Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.error.code", 
description = "Error code for the message when posting to AWS Kinesis 
Firehose"),
+@WritesAttribute(attribute = "aws.kinesis.firehose.record.id", 
description = "Record id of the message posted to Kinesis Firehose")})
+public class PutKinesisFirehose extends AbstractKinesisFirehoseProcessor {
+
+/**
+ * Kinesis put record response error message
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_MESSAGE = 
"aws.kinesis.firehose.error.message";
+
+/**
+ * Kinesis put record response error code
+ */
+public static final String AWS_KINESIS_FIREHOSE_ERROR_CODE = 
"aws.kinesis.firehose.error.code";
+
+/**
+ * Kinesis put record response record id
+ */
+public static final String AWS_KINESIS_FIREHOSE_RECORD_ID = 
"aws.kinesis.firehose.record.id";
+
+public static final List properties = 
Collections.unmodifiableList(
+Arrays.asList(KINESIS_FIREHOSE_DELIVERY_STREAM_NAME, 
MAX_BUFFER_INTERVAL,
+  MAX_BUFFER_SIZE, BATCH_SIZE, REGION, ACCESS_KEY, 
SECRET_KEY, CREDENTIALS_FILE, AWS_CREDENTIALS_PROVIDER_SERVICE, TIMEOUT,
+  PROXY_HOST,PROXY_HOST_PORT));
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@Override
+protected Collection customValidate(final 
ValidationContext validationContext) {
--- End diff --

With the alternative validators specified above, it should be possibl

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread apiri

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53845121
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/firehose/AbstractKinesisFirehoseProcessor.java
 ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor;
+
+import com.amazonaws.ClientConfiguration;
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.services.kinesisfirehose.AmazonKinesisFirehoseClient;
+
+/**
+ * This class provides processor the base class for kinesis firehose
+ */
+public abstract class AbstractKinesisFirehoseProcessor extends 
AbstractAWSCredentialsProviderProcessor {
+
+public static final PropertyDescriptor 
KINESIS_FIREHOSE_DELIVERY_STREAM_NAME = new PropertyDescriptor.Builder()
+.name("Amazon Kinesis Firehose Delivery Stream Name")
+.description("The name of kinesis firehose delivery stream")
+.expressionLanguageSupported(false)
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor MAX_BUFFER_INTERVAL = new 
PropertyDescriptor.Builder()
+.name("Max Buffer Interval")
+.description("Buffering interval for messages (between 60 and 
900 seconds).")
+.defaultValue("60")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.sensitive(false)
+.build();
+
+public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
+.name("Max Buffer Size (MB)")
+.description("Buffering size for messages (between 1MB and 
128MB).")
+.defaultValue("128")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.sensitive(false)
+.build();
+
+public static final PropertyDescriptor BATCH_SIZE = new 
PropertyDescriptor.Builder()
--- End diff --

I see this is used, but again, consider the aforementioned validator to get 
a range and help remove the unneeded customValidate


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread apiri

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53844969
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/firehose/AbstractKinesisFirehoseProcessor.java
 ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor;
+
+import com.amazonaws.ClientConfiguration;
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.services.kinesisfirehose.AmazonKinesisFirehoseClient;
+
+/**
+ * This class provides processor the base class for kinesis firehose
+ */
+public abstract class AbstractKinesisFirehoseProcessor extends 
AbstractAWSCredentialsProviderProcessor {
+
+public static final PropertyDescriptor 
KINESIS_FIREHOSE_DELIVERY_STREAM_NAME = new PropertyDescriptor.Builder()
+.name("Amazon Kinesis Firehose Delivery Stream Name")
+.description("The name of kinesis firehose delivery stream")
+.expressionLanguageSupported(false)
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor MAX_BUFFER_INTERVAL = new 
PropertyDescriptor.Builder()
+.name("Max Buffer Interval")
+.description("Buffering interval for messages (between 60 and 
900 seconds).")
+.defaultValue("60")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.sensitive(false)
+.build();
+
+public static final PropertyDescriptor MAX_BUFFER_SIZE = new 
PropertyDescriptor.Builder()
--- End diff --

Same commentary as above for MAX_BUFFER_INTERVAL.  Not sure how this is 
being used.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Accessing the name of a Processor programatically

2016-02-23 Thread Richard Miskin

For my current need providing a getName() method on Processor would do the job, 
but 
maybe a more flexible solution would be to provide an expression language to 
function
to provide the display name of the component. That function could be used in a
Processor property which would, hopefully, make it explicit that the display 
name
is being relied upon. Is it possible to do something like this via expression 
language?

Aldrin’s idea of allowing the Processor name to be set via something like 
expression
language seems like it could be useful, but it’d need expression language to 
act differently
when used for the Processor name vs properties. Might that be confusing?

Cheers,
Richard


> On 21 Feb 2016, at 23:18, Matt Burgess  wrote:
> 
> I think Aldrin's points about being able to use properties for the UI are
> pretty cool, but I also think there's room for the inverse (which is what I
> think the OP meant). The name of the processor is tied to its instance
> (though not guaranteed unique between instances, and definitely not unique
> between tasks within an instance). I would think any
> property/attribute/value associated with an instance of a processor should
> be available to it (so if name has been a "UI field", it would be no
> longer). The tradeoff, then, is when you have to name your processor a
> certain way to get it function correctly. I think this should not be the
> normal NiFi way, but should be available with the caveat that the
> documentation should reflect such a dependency. Sometimes when you're given
> enough rope to hang yourself, you build a bridge with it. Other times you
> just get dead :-P
> 
> Regards,
> Matt
> 
> On Sun, Feb 21, 2016 at 3:25 PM, Aldrin Piri  wrote:
> 
>> I think the functionality is certainly useful and like the deduplication,
>> certainly a great idea to be a bit more efficient.  However, I find myself
>> a bit hesitant to drive processor configuration via the processor UI
>> fields.  At first thought, I think something along the lines of EL that
>> could make use of processor properties for UI components like name and
>> potentially notes could be helpful.  In this case, the processor UI could
>> have its name be defined as something akin to "Retrieve data from ${url}
>> every ${run.time}"
>> 
>> Obviously some caveats would be needed such as handling the case of a
>> property that is also using EL likely wouldn't work, but personally find
>> driving presentation from processor logic to be more intuitive than the
>> inverse.
>> 
>> On Sun, Feb 21, 2016 at 7:45 AM, Joe Witt  wrote:
>> 
>>> Richard,
>>> 
>>> I don't believe the processor name (as you see in the UI or through
>>> the REST API) is made available to the processor instance itself at
>>> this time.  There is no strong/obvious case I can think of quickly to
>>> restrict access to the logical name of a ConfigurableComponent like
>>> this.
>>> 
>>> Other opinions?
>>> 
>>> Thanks
>>> Joe
>>> 
>>> On Sun, Feb 21, 2016 at 8:32 AM, Richard Miskin 
>>> wrote:
 Hi,
 
 I’ve a use case where each instance of a custom Processor on my flow
>>> represents a different instance of a remote system. The message produced
>> by
>>> my Processor needs to include the remote system name, and currently I’m
>>> using a PropertyDescriptor to allow this to be configured. This means
>> that
>>> the name is effectively set twice, once to be shown in the UI and once to
>>> be used by my Processor internally. This leads to the possibility of the
>>> two values being inconsistent which is likely to confuse people in
>> future.
 
 Is there any way I can access the name of a Processor from within an
>>> AbstractProcessor?
 
 Cheers,
 Richard
 
 
>>> 
>>

[GitHub] nifi pull request: Nifi 1489 (Support for http proxy) + Nifi 1495 ...

2016-02-23 Thread apiri

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/213#discussion_r53844010
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/firehose/AbstractKinesisFirehoseProcessor.java
 ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.firehose;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor;
+
+import com.amazonaws.ClientConfiguration;
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.services.kinesisfirehose.AmazonKinesisFirehoseClient;
+
+/**
+ * This class provides processor the base class for kinesis firehose
+ */
+public abstract class AbstractKinesisFirehoseProcessor extends 
AbstractAWSCredentialsProviderProcessor {
+
+public static final PropertyDescriptor 
KINESIS_FIREHOSE_DELIVERY_STREAM_NAME = new PropertyDescriptor.Builder()
+.name("Amazon Kinesis Firehose Delivery Stream Name")
+.description("The name of kinesis firehose delivery stream")
+.expressionLanguageSupported(false)
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor MAX_BUFFER_INTERVAL = new 
PropertyDescriptor.Builder()
+.name("Max Buffer Interval")
+.description("Buffering interval for messages (between 60 and 
900 seconds).")
+.defaultValue("60")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
--- End diff --

I don't see this property being passed through to the associated 
KinesisFirehoseClient.  Am I overlooking something?

If so, you should prefer the usage of 
StandardValidators#createLongValidator to create a bounded range.  This would 
let you then not have to do your customValidate logic.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] Release Apache NiFi 0.5.1 (RC1)

2016-02-23 Thread Mark Payne

+1 (binding) Release this package as nifi-0.5.1

Verified hashes & build w/ unit tests & contrib work. Verified that the initial 
issue that
caused the 0.5.1 release (NIFI-1527) was addressed.

-Mark


> On Feb 23, 2016, at 12:35 PM, Joe Witt  wrote:
> 
> +1 (binding) Release this package as nifi-0.5.1
> 
> Ran through all steps outlined for verification and validated numerous
> JIRAs on the release and confirmed the finding Sean had for the
> sources/javadocs has been resolved.
> 
> Thanks!
> 
> On Tue, Feb 23, 2016 at 7:27 AM, Brian Ghigiarelli  
> wrote:
>> Upgrading to JDK 8 u74 works like a charm - everything builds and tests to
>> success. The fan is a lot quieter now too. Thanks for the pointer on the
>> JDK version!
>> 
>> On Tue, Feb 23, 2016 at 7:08 AM Brian Ghigiarelli 
>> wrote:
>> 
>>> Thanks, Tony!
>>> 
>>> Oracle JDK 8 u45 (1.8.0_45)
>>> 
>>> At this point, I've tried the following MAVEN_OPTS for "mvn -T C2.0 clean
>>> install" and "mvn -T C4.0 clean install"
>>> 
>>> -Xms1024m -Xmx3076m
>>> -Xms128m -Xmx4096m
>>> -Xmx1024m -Xmx4096m -XX:+UseG1GC -XX:+UseStringDeduplication
>>> -Xmx4096m -XX:-UseGCOverheadLimit -XX:+UseG1GC
>>> -Xmx3g -XX:-UseGCOverheadLimit
>>> 
>>> All eventually fail in the same spot:
>>> 
>>> Tests run: 139, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 154.539
>>> sec <<< FAILURE! - in TestSuite
>>> testManyFilesOpened on
>>> testManyFilesOpened(org.apache.nifi.controller.repository.TestStandardProcessSession)(org.apache.nifi.controller.repository.TestStandardProcessSession)
>>> Time elapsed: 59.672 sec  <<< FAILURE!
>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>> at
>>> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:61)
>>> at
>>> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:46)
>>> at
>>> org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2009)
>>> at
>>> org.apache.nifi.controller.repository.TestStandardProcessSession.testManyFilesOpened(TestStandardProcessSession.java:775)
>>> 
>>> This is running on a MacBook Pro OS X 10.11.3 (El Capitan) with a 3GHz i7
>>> processor and 16GB memory.
>>> 
>>> On Tue, Feb 23, 2016 at 6:35 AM Tony Kurc  wrote:
>>> 
 often that error is because the heap is overcommitted What version of
 the JDK are you attempting to build with, and what MAVEN_OPTS are you
 using? I built successfully with MAVEN_OPTS="-Xms1024m -Xmx3076m".
 
 On Tue, Feb 23, 2016 at 6:04 AM, Brian Ghigiarelli 
 wrote:
 
> Hi Pierre,
> 
> I'm hitting the same issues on Mac with the master branch (specifically
> while it runs testManyOpenedFiles in nifi-framework-core). Been tuning
 Xmx,
> ulimits, and G1GC to no avail. Perhaps we could continue that
 investigation
> on a separate non-voting thread.
> 
> Thanks,
> Brian
> 
> On Tue, Feb 23, 2016 at 5:53 AM Pierre Villard <
> pierre.villard...@gmail.com>
> wrote:
> 
>> Hi all,
>> 
>> I got some issues when building the version (on Windows):
>> - with 'mvn clean install -Pcontrib-check', I got GC overhead limit
 error
>> on unit tests in framework-core (I didn't manage to get it working by
>> changing options in Maven but I guess I missed something with my own
> local
>> environment)
>> - with 'mvn clean install -Pcontrib-check -DskipTests', I got the
> following
>> error : [ERROR] Failed to execute goal
>> org.apache.rat:apache-rat-plugin:0.11:check (default) on project
>> nifi-standard-processors: Too many files with unapproved license: 1
 See
> RAT
>> report (It comes from
>> 
>> 
> 
 nifi-0.5.1/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestEncryptContent/unsalted_128_raw.enc)
>> - with 'mvn clean install -DskipTests', it is OK.
>> 
>> Otherwise, as far as I looked, it seems OK to me.
>> 
>> 
>> 
>> 
>> 2016-02-23 5:40 GMT+01:00 Tony Kurc :
>> 
>>> Hello,
>>> I am pleased to be calling this vote for the source release of
 Apache
>> NiFi
>>> nifi-0.5.1.
>>> 
>>> The source zip, including signatures, digests, etc. can be found at:
>>> 
 https://repository.apache.org/content/repositories/orgapachenifi-1075
>>> 
>>> The Git tag is nifi-0.5.1-RC1
>>> The Git commit ID is 1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
>>> 
>>> 
>> 
> 
 https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
>>> 
>>> This release candidate is a branch off of support/nifi-0.5.x at
>>> 2a459fe21c708c8316dd255c3dc7f8254e7940f1.
>>> 
>>> Checksums of nifi-0.5.1-source-release.zip:
>>> MD5: 02b920f226643412cc9ff5f7103fa156
>>> SHA1: 7109ea477559c0c0eadb93e78a8c99e765d26bce
>>> 
>>> Release artifacts are signed with the following key:
>>>

Re: [VOTE] Release Apache NiFi 0.5.1 (RC1)

2016-02-23 Thread Joe Witt

+1 (binding) Release this package as nifi-0.5.1

Ran through all steps outlined for verification and validated numerous
JIRAs on the release and confirmed the finding Sean had for the
sources/javadocs has been resolved.

Thanks!

On Tue, Feb 23, 2016 at 7:27 AM, Brian Ghigiarelli  wrote:
> Upgrading to JDK 8 u74 works like a charm - everything builds and tests to
> success. The fan is a lot quieter now too. Thanks for the pointer on the
> JDK version!
>
> On Tue, Feb 23, 2016 at 7:08 AM Brian Ghigiarelli 
> wrote:
>
>> Thanks, Tony!
>>
>> Oracle JDK 8 u45 (1.8.0_45)
>>
>> At this point, I've tried the following MAVEN_OPTS for "mvn -T C2.0 clean
>> install" and "mvn -T C4.0 clean install"
>>
>> -Xms1024m -Xmx3076m
>> -Xms128m -Xmx4096m
>> -Xmx1024m -Xmx4096m -XX:+UseG1GC -XX:+UseStringDeduplication
>> -Xmx4096m -XX:-UseGCOverheadLimit -XX:+UseG1GC
>> -Xmx3g -XX:-UseGCOverheadLimit
>>
>> All eventually fail in the same spot:
>>
>> Tests run: 139, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 154.539
>> sec <<< FAILURE! - in TestSuite
>> testManyFilesOpened on
>> testManyFilesOpened(org.apache.nifi.controller.repository.TestStandardProcessSession)(org.apache.nifi.controller.repository.TestStandardProcessSession)
>>  Time elapsed: 59.672 sec  <<< FAILURE!
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>> at
>> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:61)
>> at
>> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:46)
>> at
>> org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2009)
>> at
>> org.apache.nifi.controller.repository.TestStandardProcessSession.testManyFilesOpened(TestStandardProcessSession.java:775)
>>
>> This is running on a MacBook Pro OS X 10.11.3 (El Capitan) with a 3GHz i7
>> processor and 16GB memory.
>>
>> On Tue, Feb 23, 2016 at 6:35 AM Tony Kurc  wrote:
>>
>>> often that error is because the heap is overcommitted What version of
>>> the JDK are you attempting to build with, and what MAVEN_OPTS are you
>>> using? I built successfully with MAVEN_OPTS="-Xms1024m -Xmx3076m".
>>>
>>> On Tue, Feb 23, 2016 at 6:04 AM, Brian Ghigiarelli 
>>> wrote:
>>>
>>> > Hi Pierre,
>>> >
>>> > I'm hitting the same issues on Mac with the master branch (specifically
>>> > while it runs testManyOpenedFiles in nifi-framework-core). Been tuning
>>> Xmx,
>>> > ulimits, and G1GC to no avail. Perhaps we could continue that
>>> investigation
>>> > on a separate non-voting thread.
>>> >
>>> > Thanks,
>>> > Brian
>>> >
>>> > On Tue, Feb 23, 2016 at 5:53 AM Pierre Villard <
>>> > pierre.villard...@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi all,
>>> > >
>>> > > I got some issues when building the version (on Windows):
>>> > > - with 'mvn clean install -Pcontrib-check', I got GC overhead limit
>>> error
>>> > > on unit tests in framework-core (I didn't manage to get it working by
>>> > > changing options in Maven but I guess I missed something with my own
>>> > local
>>> > > environment)
>>> > > - with 'mvn clean install -Pcontrib-check -DskipTests', I got the
>>> > following
>>> > > error : [ERROR] Failed to execute goal
>>> > > org.apache.rat:apache-rat-plugin:0.11:check (default) on project
>>> > > nifi-standard-processors: Too many files with unapproved license: 1
>>> See
>>> > RAT
>>> > > report (It comes from
>>> > >
>>> > >
>>> >
>>> nifi-0.5.1/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestEncryptContent/unsalted_128_raw.enc)
>>> > > - with 'mvn clean install -DskipTests', it is OK.
>>> > >
>>> > > Otherwise, as far as I looked, it seems OK to me.
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > 2016-02-23 5:40 GMT+01:00 Tony Kurc :
>>> > >
>>> > > > Hello,
>>> > > > I am pleased to be calling this vote for the source release of
>>> Apache
>>> > > NiFi
>>> > > > nifi-0.5.1.
>>> > > >
>>> > > > The source zip, including signatures, digests, etc. can be found at:
>>> > > >
>>> https://repository.apache.org/content/repositories/orgapachenifi-1075
>>> > > >
>>> > > > The Git tag is nifi-0.5.1-RC1
>>> > > > The Git commit ID is 1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
>>> > > >
>>> > > >
>>> > >
>>> >
>>> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
>>> > > >
>>> > > > This release candidate is a branch off of support/nifi-0.5.x at
>>> > > > 2a459fe21c708c8316dd255c3dc7f8254e7940f1.
>>> > > >
>>> > > > Checksums of nifi-0.5.1-source-release.zip:
>>> > > > MD5: 02b920f226643412cc9ff5f7103fa156
>>> > > > SHA1: 7109ea477559c0c0eadb93e78a8c99e765d26bce
>>> > > >
>>> > > > Release artifacts are signed with the following key:
>>> > > > https://people.apache.org/keys/committer/tkurc.asc
>>> > > >
>>> > > > KEYS file available here:
>>> > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
>>> > > >
>>> > > > 11 issues were closed/resolved for this release:
>>> > > >
>>> > > >
>>> > >
>>> >
>>> https://issues.apache.org/ji

[GitHub] nifi pull request: NiFi-1481 Enhancement[ nifi.sh env]

2016-02-23 Thread trkurc

Github user trkurc commented on the pull request:

https://github.com/apache/nifi/pull/218#issuecomment-187771732
  
I looked it over and gave it a try, I don't have any substantive comments 
yet, this was queued behind getting 0.5.1 out. I think this is a good candidate 
for 0.6.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Problems with links on the Contributors Guide

2016-02-23 Thread Pierre Villard

I think you have to add the following code in the XML file you created from
the highlighted area of code concerning checkstyle in the pom file :





*http://www.puppycrawl.com/dtds/configuration_1_3.dtd
">*


...

(it may be possible to proceed differently but this is what I did on my
side)

Pierre


2016-02-23 17:00 GMT+01:00 DAVID SMITH :

> Pierre
> I have loaded Formatter file as you suggested, and that works I have tried
> both links for the checkstyle config, one gives me a 404 error the other
> takes me to the NiFi pom file and highlights a area of code which is a
> maven checkstyle plugin, I tried saving it as an xml file and importing it
> but again I got  'Document is invalid - No grammer found' error.
> Dave
>
> On Tuesday, 23 February 2016, 15:12, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
>
>  Hi David,
>
> Regarding the formatter file, it is supposed to be loaded in Preferences /
> Java  / Code style / Formatter.
>
> For the checkstyle configuration, if I remember correctly there are two
> links pointing to the configuration file but only one is valid. Did you
> manage to add the checkstyle conf?
>
> Pierre
>
> 2016-02-23 16:02 GMT+01:00 DAVID SMITH :
>
> > Hi
> > As I am looking to submit code I looked on the Contributors Guide at the
> > Code Style section.I am using Eclipse Mars and I loaded the Checksytyle
> > plugin, I downloaded the eclipse-formatter.xml from the link, but it will
> > not load into the Check Configuration panel in Eclipse, I get the
> following
> > error:Unable to parse configuration stream - Document is invalid: no
> > grammar found.:12:10
> > Has anyone seen this error before and managed to overcome it?
> > Also under the Eclipse Users heading the 'checkstyle rules' hyperlink
> > always returns error 404.
> > ThanksDave
>
>
>
>

[GitHub] nifi pull request: Enhancement for NIFI-1045 : Add "backup suffix"...

2016-02-23 Thread PuspenduBanerjee

Github user PuspenduBanerjee commented on the pull request:

https://github.com/apache/nifi/pull/230#issuecomment-187767254
  
@trixpan any chance to review again?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NiFi-1481 Enhancement[ nifi.sh env]

2016-02-23 Thread PuspenduBanerjee

Github user PuspenduBanerjee commented on the pull request:

https://github.com/apache/nifi/pull/218#issuecomment-187766997
  
@trkurc Any chance to review out of your busy schedule yet?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Problems with links on the Contributors Guide

2016-02-23 Thread DAVID SMITH

Pierre
I have loaded Formatter file as you suggested, and that works I have tried both 
links for the checkstyle config, one gives me a 404 error the other takes me to 
the NiFi pom file and highlights a area of code which is a maven checkstyle 
plugin, I tried saving it as an xml file and importing it but again I got  
'Document is invalid - No grammer found' error.
Dave 

On Tuesday, 23 February 2016, 15:12, Pierre Villard 
 wrote:

 Hi David,

Regarding the formatter file, it is supposed to be loaded in Preferences /
Java  / Code style / Formatter.

For the checkstyle configuration, if I remember correctly there are two
links pointing to the configuration file but only one is valid. Did you
manage to add the checkstyle conf?

Pierre

2016-02-23 16:02 GMT+01:00 DAVID SMITH :

> Hi
> As I am looking to submit code I looked on the Contributors Guide at the
> Code Style section.I am using Eclipse Mars and I loaded the Checksytyle
> plugin, I downloaded the eclipse-formatter.xml from the link, but it will
> not load into the Check Configuration panel in Eclipse, I get the following
> error:Unable to parse configuration stream - Document is invalid: no
> grammar found.:12:10
> Has anyone seen this error before and managed to overcome it?
> Also under the Eclipse Users heading the 'checkstyle rules' hyperlink
> always returns error 404.
> ThanksDave

Regarding execution of Map Reduce Jobs with Apache NIFI

2016-02-23 Thread shashank_kiett

Hi,

 

I want to configure a map reduce job in Apache NIFI as a processor. The
scenario for which this job  developed  is as below :

 

There are two files: 

 

1.   User_data having tab separated data like  userid
username   movieidrating

2.   Movie_data having | separated data like
movieid|movie_name

 

Requirement is :

 

To get movie name and it's aggregated rating in one
resultant file.

 

Used approach for now [Step by step]:



1.   Used ExecuteCommandScript processor with using shell script to load
and fetch data from HIVE.

2.   In  shell script I have written SQL queries for loading and
fetching data then output data was written on disk by using putFile
processor. 

 

Please suggest,

 

If I opted right approach [As I think ExecuteSQL processor should be used
for execution of SQL queries on HIVE but I do not know What is DB connection
string for it ]?

 

what is best approach for it?

 

Thanks with regards

Shashank Tiwari

Re: Problems with links on the Contributors Guide

2016-02-23 Thread Pierre Villard

Hi David,

Regarding the formatter file, it is supposed to be loaded in Preferences /
Java  / Code style / Formatter.

For the checkstyle configuration, if I remember correctly there are two
links pointing to the configuration file but only one is valid. Did you
manage to add the checkstyle conf?

Pierre

2016-02-23 16:02 GMT+01:00 DAVID SMITH :

> Hi
> As I am looking to submit code I looked on the Contributors Guide at the
> Code Style section.I am using Eclipse Mars and I loaded the Checksytyle
> plugin, I downloaded the eclipse-formatter.xml from the link, but it will
> not load into the Check Configuration panel in Eclipse, I get the following
> error:Unable to parse configuration stream - Document is invalid: no
> grammar found.:12:10
> Has anyone seen this error before and managed to overcome it?
> Also under the Eclipse Users heading the 'checkstyle rules' hyperlink
> always returns error 404.
> ThanksDave

Problems with links on the Contributors Guide

2016-02-23 Thread DAVID SMITH

Hi
As I am looking to submit code I looked on the Contributors Guide at the Code 
Style section.I am using Eclipse Mars and I loaded the Checksytyle plugin, I 
downloaded the eclipse-formatter.xml from the link, but it will not load into 
the Check Configuration panel in Eclipse, I get the following error:Unable to 
parse configuration stream - Document is invalid: no grammar found.:12:10
Has anyone seen this error before and managed to overcome it?
Also under the Eclipse Users heading the 'checkstyle rules' hyperlink always 
returns error 404.
ThanksDave

Re: Extending the NiFi API

2016-02-23 Thread DAVID SMITH

Thanks for the links Joe, I will talk to the guys at work to get their thoughts 
to see if they cover everything they may require.
Dave 

On Tuesday, 23 February 2016, 14:12, Joe Witt  wrote:

 Dave,

Absolutely this is the sort of flexibility and centralized management
we're working to enable with the 1.0.0 release.  In fact recently Matt
Gilman put in a series of tickets to head in this direction.  Those
are based on roadmap/feature proposals written up a while ago.  You
can see those here :
  https://cwiki.apache.org/confluence/display/NIFI/Multi-Tentant+Dataflow
  https://cwiki.apache.org/confluence/display/NIFI/Support+Authorizer+API

There are centralized security/policy management services including a
couple Apache projects and we'll delegate calls to them.

Take a look and see how they map to your thinking and please feel free
to suggest ideas or ask questions to ensure it is shaped in a way that
helps you move forward.  Great time to have these discussions.

Thanks
Joe

On Tue, Feb 23, 2016 at 8:55 AM, DAVID SMITH  wrote:
> Hi
> At work we run many instances of NiFi and other associated data applications, 
> with more instances of NiFi being planned it is becoming impracticable to use 
> the UI's to add, delete users or amend their roles. We have developed an in 
> house tool which controls these associated applications via their  API's 
> .Could the NiFi API be extended to allow users to be added/removed or allow 
> their roles to be updated. Thereby allowing us centrally manage many 
> instances at once.
> Dave

Re: Extending the NiFi API

2016-02-23 Thread Joe Witt

Dave,

Absolutely this is the sort of flexibility and centralized management
we're working to enable with the 1.0.0 release.  In fact recently Matt
Gilman put in a series of tickets to head in this direction.  Those
are based on roadmap/feature proposals written up a while ago.  You
can see those here :
  https://cwiki.apache.org/confluence/display/NIFI/Multi-Tentant+Dataflow
  https://cwiki.apache.org/confluence/display/NIFI/Support+Authorizer+API

There are centralized security/policy management services including a
couple Apache projects and we'll delegate calls to them.

Take a look and see how they map to your thinking and please feel free
to suggest ideas or ask questions to ensure it is shaped in a way that
helps you move forward.  Great time to have these discussions.

Thanks
Joe

On Tue, Feb 23, 2016 at 8:55 AM, DAVID SMITH  wrote:
> Hi
> At work we run many instances of NiFi and other associated data applications, 
> with more instances of NiFi being planned it is becoming impracticable to use 
> the UI's to add, delete users or amend their roles. We have developed an in 
> house tool which controls these associated applications via their  API's 
> .Could the NiFi API be extended to allow users to be added/removed or allow 
> their roles to be updated. Thereby allowing us centrally manage many 
> instances at once.
> Dave

Extending the NiFi API

2016-02-23 Thread DAVID SMITH

Hi
At work we run many instances of NiFi and other associated data applications, 
with more instances of NiFi being planned it is becoming impracticable to use 
the UI's to add, delete users or amend their roles. We have developed an in 
house tool which controls these associated applications via their  API's .Could 
the NiFi API be extended to allow users to be added/removed or allow their 
roles to be updated. Thereby allowing us centrally manage many instances at 
once.
Dave

Re: [VOTE] Release Apache NiFi 0.5.1 (RC1)

2016-02-23 Thread Brian Ghigiarelli

Upgrading to JDK 8 u74 works like a charm - everything builds and tests to
success. The fan is a lot quieter now too. Thanks for the pointer on the
JDK version!

On Tue, Feb 23, 2016 at 7:08 AM Brian Ghigiarelli 
wrote:

> Thanks, Tony!
>
> Oracle JDK 8 u45 (1.8.0_45)
>
> At this point, I've tried the following MAVEN_OPTS for "mvn -T C2.0 clean
> install" and "mvn -T C4.0 clean install"
>
> -Xms1024m -Xmx3076m
> -Xms128m -Xmx4096m
> -Xmx1024m -Xmx4096m -XX:+UseG1GC -XX:+UseStringDeduplication
> -Xmx4096m -XX:-UseGCOverheadLimit -XX:+UseG1GC
> -Xmx3g -XX:-UseGCOverheadLimit
>
> All eventually fail in the same spot:
>
> Tests run: 139, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 154.539
> sec <<< FAILURE! - in TestSuite
> testManyFilesOpened on
> testManyFilesOpened(org.apache.nifi.controller.repository.TestStandardProcessSession)(org.apache.nifi.controller.repository.TestStandardProcessSession)
>  Time elapsed: 59.672 sec  <<< FAILURE!
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at
> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:61)
> at
> org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:46)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2009)
> at
> org.apache.nifi.controller.repository.TestStandardProcessSession.testManyFilesOpened(TestStandardProcessSession.java:775)
>
> This is running on a MacBook Pro OS X 10.11.3 (El Capitan) with a 3GHz i7
> processor and 16GB memory.
>
> On Tue, Feb 23, 2016 at 6:35 AM Tony Kurc  wrote:
>
>> often that error is because the heap is overcommitted What version of
>> the JDK are you attempting to build with, and what MAVEN_OPTS are you
>> using? I built successfully with MAVEN_OPTS="-Xms1024m -Xmx3076m".
>>
>> On Tue, Feb 23, 2016 at 6:04 AM, Brian Ghigiarelli 
>> wrote:
>>
>> > Hi Pierre,
>> >
>> > I'm hitting the same issues on Mac with the master branch (specifically
>> > while it runs testManyOpenedFiles in nifi-framework-core). Been tuning
>> Xmx,
>> > ulimits, and G1GC to no avail. Perhaps we could continue that
>> investigation
>> > on a separate non-voting thread.
>> >
>> > Thanks,
>> > Brian
>> >
>> > On Tue, Feb 23, 2016 at 5:53 AM Pierre Villard <
>> > pierre.villard...@gmail.com>
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > I got some issues when building the version (on Windows):
>> > > - with 'mvn clean install -Pcontrib-check', I got GC overhead limit
>> error
>> > > on unit tests in framework-core (I didn't manage to get it working by
>> > > changing options in Maven but I guess I missed something with my own
>> > local
>> > > environment)
>> > > - with 'mvn clean install -Pcontrib-check -DskipTests', I got the
>> > following
>> > > error : [ERROR] Failed to execute goal
>> > > org.apache.rat:apache-rat-plugin:0.11:check (default) on project
>> > > nifi-standard-processors: Too many files with unapproved license: 1
>> See
>> > RAT
>> > > report (It comes from
>> > >
>> > >
>> >
>> nifi-0.5.1/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestEncryptContent/unsalted_128_raw.enc)
>> > > - with 'mvn clean install -DskipTests', it is OK.
>> > >
>> > > Otherwise, as far as I looked, it seems OK to me.
>> > >
>> > >
>> > >
>> > >
>> > > 2016-02-23 5:40 GMT+01:00 Tony Kurc :
>> > >
>> > > > Hello,
>> > > > I am pleased to be calling this vote for the source release of
>> Apache
>> > > NiFi
>> > > > nifi-0.5.1.
>> > > >
>> > > > The source zip, including signatures, digests, etc. can be found at:
>> > > >
>> https://repository.apache.org/content/repositories/orgapachenifi-1075
>> > > >
>> > > > The Git tag is nifi-0.5.1-RC1
>> > > > The Git commit ID is 1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
>> > > >
>> > > >
>> > >
>> >
>> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
>> > > >
>> > > > This release candidate is a branch off of support/nifi-0.5.x at
>> > > > 2a459fe21c708c8316dd255c3dc7f8254e7940f1.
>> > > >
>> > > > Checksums of nifi-0.5.1-source-release.zip:
>> > > > MD5: 02b920f226643412cc9ff5f7103fa156
>> > > > SHA1: 7109ea477559c0c0eadb93e78a8c99e765d26bce
>> > > >
>> > > > Release artifacts are signed with the following key:
>> > > > https://people.apache.org/keys/committer/tkurc.asc
>> > > >
>> > > > KEYS file available here:
>> > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
>> > > >
>> > > > 11 issues were closed/resolved for this release:
>> > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12334887
>> > > >
>> > > > Release note highlights can be found here:
>> > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.5.1
>> > > >
>> > > > The vote will be open for 72 hours.
>> > > > Please download the release candidate and evaluate the necessary
>> items
>> > > > including checking hashes, signatu

Re: [VOTE] Release Apache NiFi 0.5.1 (RC1)

2016-02-23 Thread Brian Ghigiarelli

Thanks, Tony!

Oracle JDK 8 u45 (1.8.0_45)

At this point, I've tried the following MAVEN_OPTS for "mvn -T C2.0 clean
install" and "mvn -T C4.0 clean install"

-Xms1024m -Xmx3076m
-Xms128m -Xmx4096m
-Xmx1024m -Xmx4096m -XX:+UseG1GC -XX:+UseStringDeduplication
-Xmx4096m -XX:-UseGCOverheadLimit -XX:+UseG1GC
-Xmx3g -XX:-UseGCOverheadLimit

All eventually fail in the same spot:

Tests run: 139, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 154.539
sec <<< FAILURE! - in TestSuite
testManyFilesOpened on
testManyFilesOpened(org.apache.nifi.controller.repository.TestStandardProcessSession)(org.apache.nifi.controller.repository.TestStandardProcessSession)
 Time elapsed: 59.672 sec  <<< FAILURE!
java.lang.OutOfMemoryError: GC overhead limit exceeded
at
org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:61)
at
org.apache.nifi.stream.io.BufferedOutputStream.(BufferedOutputStream.java:46)
at
org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2009)
at
org.apache.nifi.controller.repository.TestStandardProcessSession.testManyFilesOpened(TestStandardProcessSession.java:775)

This is running on a MacBook Pro OS X 10.11.3 (El Capitan) with a 3GHz i7
processor and 16GB memory.

On Tue, Feb 23, 2016 at 6:35 AM Tony Kurc  wrote:

> often that error is because the heap is overcommitted What version of
> the JDK are you attempting to build with, and what MAVEN_OPTS are you
> using? I built successfully with MAVEN_OPTS="-Xms1024m -Xmx3076m".
>
> On Tue, Feb 23, 2016 at 6:04 AM, Brian Ghigiarelli 
> wrote:
>
> > Hi Pierre,
> >
> > I'm hitting the same issues on Mac with the master branch (specifically
> > while it runs testManyOpenedFiles in nifi-framework-core). Been tuning
> Xmx,
> > ulimits, and G1GC to no avail. Perhaps we could continue that
> investigation
> > on a separate non-voting thread.
> >
> > Thanks,
> > Brian
> >
> > On Tue, Feb 23, 2016 at 5:53 AM Pierre Villard <
> > pierre.villard...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I got some issues when building the version (on Windows):
> > > - with 'mvn clean install -Pcontrib-check', I got GC overhead limit
> error
> > > on unit tests in framework-core (I didn't manage to get it working by
> > > changing options in Maven but I guess I missed something with my own
> > local
> > > environment)
> > > - with 'mvn clean install -Pcontrib-check -DskipTests', I got the
> > following
> > > error : [ERROR] Failed to execute goal
> > > org.apache.rat:apache-rat-plugin:0.11:check (default) on project
> > > nifi-standard-processors: Too many files with unapproved license: 1 See
> > RAT
> > > report (It comes from
> > >
> > >
> >
> nifi-0.5.1/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestEncryptContent/unsalted_128_raw.enc)
> > > - with 'mvn clean install -DskipTests', it is OK.
> > >
> > > Otherwise, as far as I looked, it seems OK to me.
> > >
> > >
> > >
> > >
> > > 2016-02-23 5:40 GMT+01:00 Tony Kurc :
> > >
> > > > Hello,
> > > > I am pleased to be calling this vote for the source release of Apache
> > > NiFi
> > > > nifi-0.5.1.
> > > >
> > > > The source zip, including signatures, digests, etc. can be found at:
> > > >
> https://repository.apache.org/content/repositories/orgapachenifi-1075
> > > >
> > > > The Git tag is nifi-0.5.1-RC1
> > > > The Git commit ID is 1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
> > > >
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
> > > >
> > > > This release candidate is a branch off of support/nifi-0.5.x at
> > > > 2a459fe21c708c8316dd255c3dc7f8254e7940f1.
> > > >
> > > > Checksums of nifi-0.5.1-source-release.zip:
> > > > MD5: 02b920f226643412cc9ff5f7103fa156
> > > > SHA1: 7109ea477559c0c0eadb93e78a8c99e765d26bce
> > > >
> > > > Release artifacts are signed with the following key:
> > > > https://people.apache.org/keys/committer/tkurc.asc
> > > >
> > > > KEYS file available here:
> > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > >
> > > > 11 issues were closed/resolved for this release:
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12334887
> > > >
> > > > Release note highlights can be found here:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.5.1
> > > >
> > > > The vote will be open for 72 hours.
> > > > Please download the release candidate and evaluate the necessary
> items
> > > > including checking hashes, signatures, build from source, and test.
> > Then
> > > > please vote:
> > > >
> > > > [ ] +1 Release this package as nifi-0.5.1
> > > > [ ] +0 no opinion
> > > > [ ] -1 Do not release this package because because...
> > > >
> > > > Thanks again for helping with this release!
> > > >
> > > > Tony
> > > >
> > >
> >
>

Re: [VOTE] Release Apache NiFi 0.5.1 (RC1)

2016-02-23 Thread Tony Kurc

often that error is because the heap is overcommitted What version of
the JDK are you attempting to build with, and what MAVEN_OPTS are you
using? I built successfully with MAVEN_OPTS="-Xms1024m -Xmx3076m".

On Tue, Feb 23, 2016 at 6:04 AM, Brian Ghigiarelli 
wrote:

> Hi Pierre,
>
> I'm hitting the same issues on Mac with the master branch (specifically
> while it runs testManyOpenedFiles in nifi-framework-core). Been tuning Xmx,
> ulimits, and G1GC to no avail. Perhaps we could continue that investigation
> on a separate non-voting thread.
>
> Thanks,
> Brian
>
> On Tue, Feb 23, 2016 at 5:53 AM Pierre Villard <
> pierre.villard...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I got some issues when building the version (on Windows):
> > - with 'mvn clean install -Pcontrib-check', I got GC overhead limit error
> > on unit tests in framework-core (I didn't manage to get it working by
> > changing options in Maven but I guess I missed something with my own
> local
> > environment)
> > - with 'mvn clean install -Pcontrib-check -DskipTests', I got the
> following
> > error : [ERROR] Failed to execute goal
> > org.apache.rat:apache-rat-plugin:0.11:check (default) on project
> > nifi-standard-processors: Too many files with unapproved license: 1 See
> RAT
> > report (It comes from
> >
> >
> nifi-0.5.1/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestEncryptContent/unsalted_128_raw.enc)
> > - with 'mvn clean install -DskipTests', it is OK.
> >
> > Otherwise, as far as I looked, it seems OK to me.
> >
> >
> >
> >
> > 2016-02-23 5:40 GMT+01:00 Tony Kurc :
> >
> > > Hello,
> > > I am pleased to be calling this vote for the source release of Apache
> > NiFi
> > > nifi-0.5.1.
> > >
> > > The source zip, including signatures, digests, etc. can be found at:
> > > https://repository.apache.org/content/repositories/orgapachenifi-1075
> > >
> > > The Git tag is nifi-0.5.1-RC1
> > > The Git commit ID is 1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
> > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
> > >
> > > This release candidate is a branch off of support/nifi-0.5.x at
> > > 2a459fe21c708c8316dd255c3dc7f8254e7940f1.
> > >
> > > Checksums of nifi-0.5.1-source-release.zip:
> > > MD5: 02b920f226643412cc9ff5f7103fa156
> > > SHA1: 7109ea477559c0c0eadb93e78a8c99e765d26bce
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/tkurc.asc
> > >
> > > KEYS file available here:
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > 11 issues were closed/resolved for this release:
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12334887
> > >
> > > Release note highlights can be found here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.5.1
> > >
> > > The vote will be open for 72 hours.
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build from source, and test.
> Then
> > > please vote:
> > >
> > > [ ] +1 Release this package as nifi-0.5.1
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because because...
> > >
> > > Thanks again for helping with this release!
> > >
> > > Tony
> > >
> >
>

Re: [VOTE] Release Apache NiFi 0.5.1 (RC1)

2016-02-23 Thread Brian Ghigiarelli

Hi Pierre,

I'm hitting the same issues on Mac with the master branch (specifically
while it runs testManyOpenedFiles in nifi-framework-core). Been tuning Xmx,
ulimits, and G1GC to no avail. Perhaps we could continue that investigation
on a separate non-voting thread.

Thanks,
Brian

On Tue, Feb 23, 2016 at 5:53 AM Pierre Villard 
wrote:

> Hi all,
>
> I got some issues when building the version (on Windows):
> - with 'mvn clean install -Pcontrib-check', I got GC overhead limit error
> on unit tests in framework-core (I didn't manage to get it working by
> changing options in Maven but I guess I missed something with my own local
> environment)
> - with 'mvn clean install -Pcontrib-check -DskipTests', I got the following
> error : [ERROR] Failed to execute goal
> org.apache.rat:apache-rat-plugin:0.11:check (default) on project
> nifi-standard-processors: Too many files with unapproved license: 1 See RAT
> report (It comes from
>
> nifi-0.5.1/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestEncryptContent/unsalted_128_raw.enc)
> - with 'mvn clean install -DskipTests', it is OK.
>
> Otherwise, as far as I looked, it seems OK to me.
>
>
>
>
> 2016-02-23 5:40 GMT+01:00 Tony Kurc :
>
> > Hello,
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > nifi-0.5.1.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1075
> >
> > The Git tag is nifi-0.5.1-RC1
> > The Git commit ID is 1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
> >
> > This release candidate is a branch off of support/nifi-0.5.x at
> > 2a459fe21c708c8316dd255c3dc7f8254e7940f1.
> >
> > Checksums of nifi-0.5.1-source-release.zip:
> > MD5: 02b920f226643412cc9ff5f7103fa156
> > SHA1: 7109ea477559c0c0eadb93e78a8c99e765d26bce
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/tkurc.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 11 issues were closed/resolved for this release:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12334887
> >
> > Release note highlights can be found here:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.5.1
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test.  Then
> > please vote:
> >
> > [ ] +1 Release this package as nifi-0.5.1
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because because...
> >
> > Thanks again for helping with this release!
> >
> > Tony
> >
>

Re: [VOTE] Release Apache NiFi 0.5.1 (RC1)

2016-02-23 Thread Pierre Villard

Hi all,

I got some issues when building the version (on Windows):
- with 'mvn clean install -Pcontrib-check', I got GC overhead limit error
on unit tests in framework-core (I didn't manage to get it working by
changing options in Maven but I guess I missed something with my own local
environment)
- with 'mvn clean install -Pcontrib-check -DskipTests', I got the following
error : [ERROR] Failed to execute goal
org.apache.rat:apache-rat-plugin:0.11:check (default) on project
nifi-standard-processors: Too many files with unapproved license: 1 See RAT
report (It comes from
nifi-0.5.1/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestEncryptContent/unsalted_128_raw.enc)
- with 'mvn clean install -DskipTests', it is OK.

Otherwise, as far as I looked, it seems OK to me.




2016-02-23 5:40 GMT+01:00 Tony Kurc :

> Hello,
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-0.5.1.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1075
>
> The Git tag is nifi-0.5.1-RC1
> The Git commit ID is 1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=1bc7bdb609606b9aa61cebad3ef2e3c87d34ea63
>
> This release candidate is a branch off of support/nifi-0.5.x at
> 2a459fe21c708c8316dd255c3dc7f8254e7940f1.
>
> Checksums of nifi-0.5.1-source-release.zip:
> MD5: 02b920f226643412cc9ff5f7103fa156
> SHA1: 7109ea477559c0c0eadb93e78a8c99e765d26bce
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/tkurc.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 11 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12334887
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.5.1
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.  Then
> please vote:
>
> [ ] +1 Release this package as nifi-0.5.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because because...
>
> Thanks again for helping with this release!
>
> Tony
>

48 matches

Mail list logo