[GitHub] nifi issue #362: NIFI-1769: added support for SSE-KMS and signature s3v4 aut...

[jvwing]

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/362
  
@miquillo , what do you think about using 
[ClientConfiguration::setSignerOverride()](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html#setSignerOverride(java.lang.String))
 to control the signature version?  One of the advantages I see to it is better 
isolation for the processor vs. any other NiFi AWS processors.  I'm a bit 
worried that one PutS3Object processor using SSE-KMS would change the settings 
other processors running at the same time.  I believe an appropriate location 
to do this would be in AbstractS3Processor::createClient().  That would allow 
the configuration code to be shared, while the configured value would remain 
specific to individual processors.

But I'm not sure I agree with configuring the version as a true/false 
setting for signature version 4.  I would recommend a list of values:

* AWS SDK default (as the default selection)
* Signature v2
* Signature v4

That leaves room for the AWS SDK default to change if/when we upgrade to a 
newer SDK, and it would allow for users to explicitly request either v4 or v2 
to match whatever features and endpoint they are using.  What do you think?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #362: NIFI-1769: added support for SSE-KMS and signature s3v4 aut...

[jvwing]

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/362
  
I now better understand that we do not need to do anything to the 
FetchS3Object processor to read KMS encrypted files.  Server-Side Encryption 
is, after all, on the server.  Strike that point.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #541: NIFI-2032 fixed 'enableControllerServices' logic

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/541
  
I merged into master but I've had to rebase and resolve conflicts 3 times 
on `0.x` branch and still have tests failing. Will continue working on that 
part in the morning. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67801359
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles 

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[joewitt]

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67800869
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles 

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67800576
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles 

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[joewitt]

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67799555
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles 

[GitHub] nifi issue #517: NIFI-1994: Fixed issues with controller services and templa...

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/517
  
Looks good @markap14 

Had one minor issue ensuring controller services are copied as part of a 
process group dto. I tweak your commit to include the additionally 
functionality.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #517: NIFI-1994: Fixed issues with controller services and...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/517


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67799164
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles 

[GitHub] nifi pull request #541: NIFI-2032 fixed 'enableControllerServices' logic

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/541


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #252: NIFI-615 - Create a processor to extract WAV file character...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/252
  
@jskora were you able to rebase and open a new PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #511: NIFI-1850 - JSON-to-JSON Schema Converter Editor

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/511
  
Looks great @YolandaMDavis 

I'm merged this into master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #511: NIFI-1850 - JSON-to-JSON Schema Converter Editor

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/511


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: 0.7.0 install bug centos 7

[Nicholas Hughes]

FWIW, I "fixed" this on my system with a different workaround:

1. Don't use the "install" option.
2. Symlink /etc/init.d/nifi to /opt/nifi/current/bin/nifi.sh
3. Edit the script to use readlink and find the actual install directory:
SCRIPT_DIR=$(dirname $(readlink $0 || echo $0))
SCRIPT_NAME=$(basename $(readlink $0 || echo $0))
(from memory, so I think that should work)

Not perfect, but seemed cleaner than having two nifi scripts kicking
around...

-Nick


On Mon, Jun 20, 2016 at 7:01 PM, James Wing  wrote:

> Thanks for reporting this, Ryan, I see the same behavior.  I created a JIRA
> for this issue (https://issues.apache.org/jira/browse/NIFI-2063).  If you
> need a workaround, you can edit /etc/init.d/nifi to hard-code the path to
> the script directory:
>
> SCRIPT_DIR=/opt/nifi/current/bin
> SCRIPT_NAME=$(basename "$0")
> PROGNAME=nifi.sh
>
> . "$SCRIPT_DIR"/nifi-env.sh
>
> ...
>
> Thanks,
>
> James
>
> On Mon, Jun 20, 2016 at 2:33 PM, Ryan H 
> wrote:
>
> > Looks like 0.7.0 doesn't install as a service quite right on linux/centos
> > 7.
> >
> > I untar'd it in:
> >  #/opt/nifi/current -> nifi-0.7.0-SNAPSHOT
> >
> > I followed these steps:
> > #/opt/nifi/current/bin/nifi.sh install
> > #chkconfig nifi on
> > #service nifi start
> >
> > That copies the nifi.sh script into /etc/init.d correctly, however it
> can't
> > start as a service because the "dirname" when starting as a service (at
> > least in my case) appears to be /etc/init.d, instead of the nifi bin dir,
> > resulting in this error:
> >
> > #$ sudo service nifi start
> > #/etc/init.d/nifi: line 28: /etc/init.d/nifi-env.sh: No such file or
> > directory
> >
> > I changed the SCRIPT_DIR variable to be my current nifi bin dir, to get
> it
> > to work.
> >
> >1. From: # SCRIPT_DIR=$(dirname "$0")
> >2. To: # SCRIPT_DIR=/opt/nifi/current/bin
> >
> >
> > Thanks,
> > Ryan
> >
>

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67783621
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles that 

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67782966
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles 

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67782856
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles 

Re: 0.7.0 install bug centos 7

[James Wing]

Thanks for reporting this, Ryan, I see the same behavior.  I created a JIRA
for this issue (https://issues.apache.org/jira/browse/NIFI-2063).  If you
need a workaround, you can edit /etc/init.d/nifi to hard-code the path to
the script directory:

SCRIPT_DIR=/opt/nifi/current/bin
SCRIPT_NAME=$(basename "$0")
PROGNAME=nifi.sh

. "$SCRIPT_DIR"/nifi-env.sh

...

Thanks,

James

On Mon, Jun 20, 2016 at 2:33 PM, Ryan H  wrote:

> Looks like 0.7.0 doesn't install as a service quite right on linux/centos
> 7.
>
> I untar'd it in:
>  #/opt/nifi/current -> nifi-0.7.0-SNAPSHOT
>
> I followed these steps:
> #/opt/nifi/current/bin/nifi.sh install
> #chkconfig nifi on
> #service nifi start
>
> That copies the nifi.sh script into /etc/init.d correctly, however it can't
> start as a service because the "dirname" when starting as a service (at
> least in my case) appears to be /etc/init.d, instead of the nifi bin dir,
> resulting in this error:
>
> #$ sudo service nifi start
> #/etc/init.d/nifi: line 28: /etc/init.d/nifi-env.sh: No such file or
> directory
>
> I changed the SCRIPT_DIR variable to be my current nifi bin dir, to get it
> to work.
>
>1. From: # SCRIPT_DIR=$(dirname "$0")
>2. To: # SCRIPT_DIR=/opt/nifi/current/bin
>
>
> Thanks,
> Ryan
>

[GitHub] nifi issue #541: NIFI-2032 fixed 'enableControllerServices' logic

[olegz]

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/541
  
@alopresto so I just fixed it and squashed it with the last commit. No 
compilation error on my end. LMK


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67780170
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles that 

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67779956
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutTCP.java
 ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.processors.standard.util.TestPutTCPCommon;
+
+public class TestPutTCP extends TestPutTCPCommon {
+
+public TestPutTCP() {
+super();
+ssl = false;
+}
+
+@Override
+public void configureProperties(String host, int port, String 
outgoingMessageDelimiter, boolean connectionPerFlowFile, boolean expectValid) {
+runner.setProperty(PutTCP.HOSTNAME, host);
+runner.setProperty(PutTCP.PORT, Integer.toString(port));
+if (outgoingMessageDelimiter != null) {
+runner.setProperty(PutTCP.OUTGOING_MESSAGE_DELIMITER, 
outgoingMessageDelimiter);
+}
+if (connectionPerFlowFile) {
+runner.setProperty(PutTCP.CONNECTION_PER_FLOWFILE, "true");
+} else {
+runner.setProperty(PutTCP.CONNECTION_PER_FLOWFILE, "false");
+}
--- End diff --

While its a style question you and I did talk about it. You are performing 
the same thing in both IF and ELSE with value being the only difference, so 
some may say its to verbose (5 lines vs 1).
That said, I do like the idea of 
```runner.setProperty(PutTCP.CONNECTION_PER_FLOWFILE, 
String.valueOf(connectionPerFlowFile))```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67779821
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles that 

[GitHub] nifi issue #541: NIFI-2032 fixed 'enableControllerServices' logic

[olegz]

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/541
  
I see, it appears that I have to reference different 
```org.apache.nifi.events.BulletinFactory```, building now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67778279
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles that 

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r6337
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutTCP.java
 ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.processors.standard.util.TestPutTCPCommon;
+
+public class TestPutTCP extends TestPutTCPCommon {
+
+public TestPutTCP() {
+super();
+ssl = false;
+}
+
+@Override
+public void configureProperties(String host, int port, String 
outgoingMessageDelimiter, boolean connectionPerFlowFile, boolean expectValid) {
+runner.setProperty(PutTCP.HOSTNAME, host);
+runner.setProperty(PutTCP.PORT, Integer.toString(port));
+if (outgoingMessageDelimiter != null) {
+runner.setProperty(PutTCP.OUTGOING_MESSAGE_DELIMITER, 
outgoingMessageDelimiter);
+}
+if (connectionPerFlowFile) {
+runner.setProperty(PutTCP.CONNECTION_PER_FLOWFILE, "true");
+} else {
+runner.setProperty(PutTCP.CONNECTION_PER_FLOWFILE, "false");
+}
--- End diff --

In general, I like the IF/ELSE block better but actually I think neither 
are appropriate here and it would better to just do 
`String.valueOf(connectionPerFlowFile)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67776824
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles 

[GitHub] nifi issue #541: NIFI-2032 fixed 'enableControllerServices' logic

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/541
  
Getting a compilation failure missing `import 
org.apache.nifi.reporting.BulletinFactory`. Did this get missed in the second 
commit?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: ControllerService Enabling and Processor onPropertyModified

[Oleg Zhurakousky]

Indeed this sounds like https://issues.apache.org/jira/browse/NIFI-2032. The PR 
for it has been out https://github.com/apache/nifi/pull/541 so if you get a 
chance to try it please let us know. Hopefully it will be merged soon.

Cheers
Oleg

On Jun 20, 2016, at 4:25 PM, Michael Moser 
> wrote:

Michael,

You may be encountering the bug NIFI-2032 [1] which exists in NiFi 0.6.1.

[1] - https://issues.apache.org/jira/browse/NIFI-2032

-- Mike



On Mon, Jun 20, 2016 at 12:20 PM, Michael D. Coon 

wrote:

All,
  Before I get too deep in submitting Jira tickets, etc. I'm wondering if
this is expected behavior. I'm using NiFi 0.6.1.

  I have a ControllerService that I reference as a service property on my
Processor. The Processor, in turn, uses the ControllerService's internal
configuration state to determine Processor output relationships. But, it
appears that at NiFi startup, I'm given the ControllerService before it is
actually enabled. When I try to invoke the  methods to get the
ControllerService's state, it fails (because it's not enabled).
  Two problems I found:
1) Logging the invocation exception for calling a disabled
ControllerService is being suppressed in this case, which caused this to
take me a full day to track down2) Why would I ever be given a service to
use without it being fully enabled?
I thought I would just block in my Processor's onPropertyModified method
until the ControllerService was enabled; but it looks like the thread
that's actually enabling the service is calling my Processor's
onPropertyModified method and the ControllerService is never enabled until
my Processor's onPropertyModified method is done.
Is this expected behavior? If so, can someone please explain the
assumptions around sending non-enabled ControllerServices to my Processor?

Mike

0.7.0 install bug centos 7

[Ryan H]

Looks like 0.7.0 doesn't install as a service quite right on linux/centos 7.

I untar'd it in:
 #/opt/nifi/current -> nifi-0.7.0-SNAPSHOT

I followed these steps:
#/opt/nifi/current/bin/nifi.sh install
#chkconfig nifi on
#service nifi start

That copies the nifi.sh script into /etc/init.d correctly, however it can't
start as a service because the "dirname" when starting as a service (at
least in my case) appears to be /etc/init.d, instead of the nifi bin dir,
resulting in this error:

#$ sudo service nifi start
#/etc/init.d/nifi: line 28: /etc/init.d/nifi-env.sh: No such file or
directory

I changed the SCRIPT_DIR variable to be my current nifi bin dir, to get it
to work.

   1. From: # SCRIPT_DIR=$(dirname "$0")
   2. To: # SCRIPT_DIR=/opt/nifi/current/bin


Thanks,
Ryan

[GitHub] nifi pull request #541: NIFI-2032 fixed 'enableControllerServices' logic

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/541#discussion_r67766144
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceProvider.java
 ---
@@ -372,98 +369,32 @@ public void enableControllerService(final 
ControllerServiceNode serviceNode) {
 
 @Override
 public void enableControllerServices(final 
Collection serviceNodes) {
-final Set servicesToEnable = new 
HashSet<>();
-// Ensure that all nodes are already disabled
-for (final ControllerServiceNode serviceNode : serviceNodes) {
-final ControllerServiceState curState = serviceNode.getState();
-if (ControllerServiceState.DISABLED.equals(curState)) {
-servicesToEnable.add(serviceNode);
-} else {
-logger.warn("Cannot enable {} because it is not disabled; 
current state is {}", serviceNode, curState);
+boolean shouldStart = true;
+
+Iterator serviceIter = 
serviceNodes.iterator();
+while (serviceIter.hasNext() && shouldStart) {
+ControllerServiceNode controllerServiceNode = 
serviceIter.next();
+List requiredServices = 
((StandardControllerServiceNode) 
controllerServiceNode).getRequiredControllerServices();
+for (ControllerServiceNode requiredService : requiredServices) 
{
+if (!requiredService.isActive() && 
!serviceNodes.contains(requiredService)) {
+shouldStart = false;
+}
 }
 }
 
-// determine the order to load the services. We have to ensure 
that if service A references service B, then B
-// is enabled first, and so on.
-final Map idToNodeMap = new 
HashMap<>();
-for (final ControllerServiceNode node : servicesToEnable) {
-idToNodeMap.put(node.getIdentifier(), node);
-}
-
-// We can have many Controller Services dependent on one another. 
We can have many of these
-// disparate lists of Controller Services that are dependent on 
one another. We refer to each
-// of these as a branch.
-final List branches = 
determineEnablingOrder(idToNodeMap);
-
-if (branches.isEmpty()) {
-logger.info("No Controller Services to enable");
-return;
-} else {
-logger.info("Will enable {} Controller Services", 
servicesToEnable.size());
-}
-
-final Set enabledNodes = 
Collections.synchronizedSet(new HashSet());
-final ExecutorService executor = 
Executors.newFixedThreadPool(Math.min(10, branches.size()), new ThreadFactory() 
{
-@Override
-public Thread newThread(final Runnable r) {
-final Thread t = 
Executors.defaultThreadFactory().newThread(r);
-t.setDaemon(true);
-t.setName("Enable Controller Services");
-return t;
-}
-});
-
-for (final List branch : branches) {
-final Runnable enableBranchRunnable = new Runnable() {
+if (shouldStart) {
+List services = new 
ArrayList<>(serviceNodes);
+Collections.sort(services, new 
Comparator() {
 @Override
-public void run() {
-logger.debug("Enabling Controller Service Branch {}", 
branch);
-
-for (final ControllerServiceNode serviceNode : branch) 
{
-try {
-if (!enabledNodes.contains(serviceNode)) {
-enabledNodes.add(serviceNode);
-
-logger.info("Enabling {}", serviceNode);
-try {
-serviceNode.verifyCanEnable();
-
processScheduler.enableControllerService(serviceNode);
-} catch (final Exception e) {
-logger.error("Failed to enable " + 
serviceNode + " due to " + e);
-if (logger.isDebugEnabled()) {
-logger.error("", e);
-}
-
-if (bulletinRepo != null) {
--- End diff --

So, I've just put _bulletinRepo_ logic back. The commit isn't squashed so 
you can see the changes in isolation.


---
If your project is set up for it, you can reply to this email and have your

[GitHub] nifi issue #541: NIFI-2032 fixed 'enableControllerServices' logic

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/541
  
Thanks @olegz . I will build locally and merge. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: ControllerService Enabling and Processor onPropertyModified

[Andy LoPresto]

I am reviewing PR 541 [1] for that bug right now.

[1] https://github.com/apache/nifi/pull/541 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jun 20, 2016, at 1:25 PM, Michael Moser  wrote:
> 
> Michael,
> 
> You may be encountering the bug NIFI-2032 [1] which exists in NiFi 0.6.1.
> 
> [1] - https://issues.apache.org/jira/browse/NIFI-2032
> 
> -- Mike
> 
> 
> 
> On Mon, Jun 20, 2016 at 12:20 PM, Michael D. Coon > wrote:
> 
>> All,
>>   Before I get too deep in submitting Jira tickets, etc. I'm wondering if
>> this is expected behavior. I'm using NiFi 0.6.1.
>> 
>>   I have a ControllerService that I reference as a service property on my
>> Processor. The Processor, in turn, uses the ControllerService's internal
>> configuration state to determine Processor output relationships. But, it
>> appears that at NiFi startup, I'm given the ControllerService before it is
>> actually enabled. When I try to invoke the  methods to get the
>> ControllerService's state, it fails (because it's not enabled).
>>   Two problems I found:
>> 1) Logging the invocation exception for calling a disabled
>> ControllerService is being suppressed in this case, which caused this to
>> take me a full day to track down2) Why would I ever be given a service to
>> use without it being fully enabled?
>> I thought I would just block in my Processor's onPropertyModified method
>> until the ControllerService was enabled; but it looks like the thread
>> that's actually enabling the service is calling my Processor's
>> onPropertyModified method and the ControllerService is never enabled until
>> my Processor's onPropertyModified method is done.
>> Is this expected behavior? If so, can someone please explain the
>> assumptions around sending non-enabled ControllerServices to my Processor?
>> 
>> Mike
>> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: ControllerService Enabling and Processor onPropertyModified

[Michael Moser]

Michael,

You may be encountering the bug NIFI-2032 [1] which exists in NiFi 0.6.1.

[1] - https://issues.apache.org/jira/browse/NIFI-2032

-- Mike



On Mon, Jun 20, 2016 at 12:20 PM, Michael D. Coon  wrote:

> All,
>Before I get too deep in submitting Jira tickets, etc. I'm wondering if
> this is expected behavior. I'm using NiFi 0.6.1.
>
>I have a ControllerService that I reference as a service property on my
> Processor. The Processor, in turn, uses the ControllerService's internal
> configuration state to determine Processor output relationships. But, it
> appears that at NiFi startup, I'm given the ControllerService before it is
> actually enabled. When I try to invoke the  methods to get the
> ControllerService's state, it fails (because it's not enabled).
>Two problems I found:
> 1) Logging the invocation exception for calling a disabled
> ControllerService is being suppressed in this case, which caused this to
> take me a full day to track down2) Why would I ever be given a service to
> use without it being fully enabled?
> I thought I would just block in my Processor's onPropertyModified method
> until the ControllerService was enabled; but it looks like the thread
> that's actually enabling the service is calling my Processor's
> onPropertyModified method and the ControllerService is never enabled until
> my Processor's onPropertyModified method is done.
> Is this expected behavior? If so, can someone please explain the
> assumptions around sending non-enabled ControllerServices to my Processor?
>
> Mike
>

[GitHub] nifi pull request #550: NIFI-1900: Verify that connection's destination is n...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/550

NIFI-1900: Verify that connection's destination is not running when t…

…rying to change destination

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-1900

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #550


commit 72679a9ca482f1647e819e40193bed477f59cb97
Author: Mark Payne 
Date:   2016-06-20T18:24:33Z

NIFI-1900: Verify that connection's destination is not running when trying 
to change destination




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #541: NIFI-2032 fixed 'enableControllerServices' logic

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/541#discussion_r67740493
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceProvider.java
 ---
@@ -372,98 +369,32 @@ public void enableControllerService(final 
ControllerServiceNode serviceNode) {
 
 @Override
 public void enableControllerServices(final 
Collection serviceNodes) {
-final Set servicesToEnable = new 
HashSet<>();
-// Ensure that all nodes are already disabled
-for (final ControllerServiceNode serviceNode : serviceNodes) {
-final ControllerServiceState curState = serviceNode.getState();
-if (ControllerServiceState.DISABLED.equals(curState)) {
-servicesToEnable.add(serviceNode);
-} else {
-logger.warn("Cannot enable {} because it is not disabled; 
current state is {}", serviceNode, curState);
+boolean shouldStart = true;
+
+Iterator serviceIter = 
serviceNodes.iterator();
+while (serviceIter.hasNext() && shouldStart) {
+ControllerServiceNode controllerServiceNode = 
serviceIter.next();
+List requiredServices = 
((StandardControllerServiceNode) 
controllerServiceNode).getRequiredControllerServices();
+for (ControllerServiceNode requiredService : requiredServices) 
{
+if (!requiredService.isActive() && 
!serviceNodes.contains(requiredService)) {
+shouldStart = false;
+}
 }
 }
 
-// determine the order to load the services. We have to ensure 
that if service A references service B, then B
-// is enabled first, and so on.
-final Map idToNodeMap = new 
HashMap<>();
-for (final ControllerServiceNode node : servicesToEnable) {
-idToNodeMap.put(node.getIdentifier(), node);
-}
-
-// We can have many Controller Services dependent on one another. 
We can have many of these
-// disparate lists of Controller Services that are dependent on 
one another. We refer to each
-// of these as a branch.
-final List branches = 
determineEnablingOrder(idToNodeMap);
-
-if (branches.isEmpty()) {
-logger.info("No Controller Services to enable");
-return;
-} else {
-logger.info("Will enable {} Controller Services", 
servicesToEnable.size());
-}
-
-final Set enabledNodes = 
Collections.synchronizedSet(new HashSet());
-final ExecutorService executor = 
Executors.newFixedThreadPool(Math.min(10, branches.size()), new ThreadFactory() 
{
-@Override
-public Thread newThread(final Runnable r) {
-final Thread t = 
Executors.defaultThreadFactory().newThread(r);
-t.setDaemon(true);
-t.setName("Enable Controller Services");
-return t;
-}
-});
-
-for (final List branch : branches) {
-final Runnable enableBranchRunnable = new Runnable() {
+if (shouldStart) {
+List services = new 
ArrayList<>(serviceNodes);
+Collections.sort(services, new 
Comparator() {
 @Override
-public void run() {
-logger.debug("Enabling Controller Service Branch {}", 
branch);
-
-for (final ControllerServiceNode serviceNode : branch) 
{
-try {
-if (!enabledNodes.contains(serviceNode)) {
-enabledNodes.add(serviceNode);
-
-logger.info("Enabling {}", serviceNode);
-try {
-serviceNode.verifyCanEnable();
-
processScheduler.enableControllerService(serviceNode);
-} catch (final Exception e) {
-logger.error("Failed to enable " + 
serviceNode + " due to " + e);
-if (logger.isDebugEnabled()) {
-logger.error("", e);
-}
-
-if (bulletinRepo != null) {
--- End diff --

Ok, we should probably maintain that behavior, correct? Not sure why it was 
commented out initially. 


---
If your project is set up for it, you can reply to this email and have your
reply 

[GitHub] nifi pull request #549: NIFI-2046 - Adding reference to @TriggerSerially in ...

[brosander]

GitHub user brosander opened a pull request:

https://github.com/apache/nifi/pull/549

NIFI-2046 - Adding reference to @TriggerSerially in Processor javadoc



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/brosander/nifi NIFI-2046

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/549.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #549


commit db95a719590c03993b1df454c5fe20658edf3d1c
Author: Bryan Rosander 
Date:   2016-06-20T18:22:02Z

NIFI-2046 - Adding reference to @TriggerSerially in Processor javadoc




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #541: NIFI-2032 fixed 'enableControllerServices' logic

[mcgilman]

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/541#discussion_r67739760
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceProvider.java
 ---
@@ -70,7 +67,7 @@
 
 private final ProcessScheduler processScheduler;
 private static final Set validDisabledMethods;
-private final BulletinRepository bulletinRepo;
+// private final BulletinRepository bulletinRepo;
--- End diff --

I would have to defer to @markap14 but it appears it was being used. See 
below.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #541: NIFI-2032 fixed 'enableControllerServices' logic

[mcgilman]

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/541#discussion_r67739510
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceProvider.java
 ---
@@ -372,98 +369,32 @@ public void enableControllerService(final 
ControllerServiceNode serviceNode) {
 
 @Override
 public void enableControllerServices(final 
Collection serviceNodes) {
-final Set servicesToEnable = new 
HashSet<>();
-// Ensure that all nodes are already disabled
-for (final ControllerServiceNode serviceNode : serviceNodes) {
-final ControllerServiceState curState = serviceNode.getState();
-if (ControllerServiceState.DISABLED.equals(curState)) {
-servicesToEnable.add(serviceNode);
-} else {
-logger.warn("Cannot enable {} because it is not disabled; 
current state is {}", serviceNode, curState);
+boolean shouldStart = true;
+
+Iterator serviceIter = 
serviceNodes.iterator();
+while (serviceIter.hasNext() && shouldStart) {
+ControllerServiceNode controllerServiceNode = 
serviceIter.next();
+List requiredServices = 
((StandardControllerServiceNode) 
controllerServiceNode).getRequiredControllerServices();
+for (ControllerServiceNode requiredService : requiredServices) 
{
+if (!requiredService.isActive() && 
!serviceNodes.contains(requiredService)) {
+shouldStart = false;
+}
 }
 }
 
-// determine the order to load the services. We have to ensure 
that if service A references service B, then B
-// is enabled first, and so on.
-final Map idToNodeMap = new 
HashMap<>();
-for (final ControllerServiceNode node : servicesToEnable) {
-idToNodeMap.put(node.getIdentifier(), node);
-}
-
-// We can have many Controller Services dependent on one another. 
We can have many of these
-// disparate lists of Controller Services that are dependent on 
one another. We refer to each
-// of these as a branch.
-final List branches = 
determineEnablingOrder(idToNodeMap);
-
-if (branches.isEmpty()) {
-logger.info("No Controller Services to enable");
-return;
-} else {
-logger.info("Will enable {} Controller Services", 
servicesToEnable.size());
-}
-
-final Set enabledNodes = 
Collections.synchronizedSet(new HashSet());
-final ExecutorService executor = 
Executors.newFixedThreadPool(Math.min(10, branches.size()), new ThreadFactory() 
{
-@Override
-public Thread newThread(final Runnable r) {
-final Thread t = 
Executors.defaultThreadFactory().newThread(r);
-t.setDaemon(true);
-t.setName("Enable Controller Services");
-return t;
-}
-});
-
-for (final List branch : branches) {
-final Runnable enableBranchRunnable = new Runnable() {
+if (shouldStart) {
+List services = new 
ArrayList<>(serviceNodes);
+Collections.sort(services, new 
Comparator() {
 @Override
-public void run() {
-logger.debug("Enabling Controller Service Branch {}", 
branch);
-
-for (final ControllerServiceNode serviceNode : branch) 
{
-try {
-if (!enabledNodes.contains(serviceNode)) {
-enabledNodes.add(serviceNode);
-
-logger.info("Enabling {}", serviceNode);
-try {
-serviceNode.verifyCanEnable();
-
processScheduler.enableControllerService(serviceNode);
-} catch (final Exception e) {
-logger.error("Failed to enable " + 
serviceNode + " due to " + e);
-if (logger.isDebugEnabled()) {
-logger.error("", e);
-}
-
-if (bulletinRepo != null) {
--- End diff --

Looks like we were creating a bulletin whenever we were unable to enable a 
Controller Service.


---
If your project is set up for it, you can reply to this email and have your
reply appear on 

[GitHub] nifi pull request #548: NIFI-2045 - Removing mqtt message from queue after a...

[brosander]

GitHub user brosander opened a pull request:

https://github.com/apache/nifi/pull/548

NIFI-2045 - Removing mqtt message from queue after after session.comit()



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/brosander/nifi NIFI-2045

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/548.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #548






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #541: NIFI-2032 fixed 'enableControllerServices' logic

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/541#discussion_r67738328
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceProvider.java
 ---
@@ -70,7 +67,7 @@
 
 private final ProcessScheduler processScheduler;
 private static final Set validDisabledMethods;
-private final BulletinRepository bulletinRepo;
+// private final BulletinRepository bulletinRepo;
--- End diff --

I removed the two commented lines (the declaration and the assignment in 
the constructor). However, I did not change the constructor contract, as it is 
used 17 times throughout the application. @mcgilman @markap14 is the bulletin 
repository work going to have additional impact, or is it safe to clean up this 
constructor universally?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Using NiFi S3 Processors with S3-Compatible Services?

[James Wing]

Is anyone using NiFi's S3 processors to work with non-Amazon, S3-compatible
services?  Would you be willing to share which service you are using and if
you have experienced compatibility issues?

Are you aware of the difference between S3's API signature versions 2 and
4?  NiFi's current default S3 signature version is version 2.  Would you be
concerned about upgrading to a more recent AWS SDK where more operations
default to version 4?

Changes in Signature Version 4
http://docs.aws.amazon.com/general/latest/gr/sigv4_changes.html


Thanks,

James

[GitHub] nifi issue #459: NIFI-1909 Adding ability to process schemaless Avro records...

[markap14]

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/459
  
@ryanpersaud thanks for getting this in! I have verified the checkstyle and 
pushed to both the 0.x and 1.0.0 baselines. Thanks again for knocking this out!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #459: NIFI-1909 Adding ability to process schemaless Avro ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/459


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67733058
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles that 

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67732780
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles that 

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67732631
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutTCP.java
 ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.nifi.processors.standard.util.TestPutTCPCommon;
+
+public class TestPutTCP extends TestPutTCPCommon {
+
+public TestPutTCP() {
+super();
+ssl = false;
+}
+
+@Override
+public void configureProperties(String host, int port, String 
outgoingMessageDelimiter, boolean connectionPerFlowFile, boolean expectValid) {
+runner.setProperty(PutTCP.HOSTNAME, host);
+runner.setProperty(PutTCP.PORT, Integer.toString(port));
+if (outgoingMessageDelimiter != null) {
+runner.setProperty(PutTCP.OUTGOING_MESSAGE_DELIMITER, 
outgoingMessageDelimiter);
+}
+if (connectionPerFlowFile) {
+runner.setProperty(PutTCP.CONNECTION_PER_FLOWFILE, "true");
+} else {
+runner.setProperty(PutTCP.CONNECTION_PER_FLOWFILE, "false");
+}
--- End diff --

This is more of a style but it appears it would be a bit more concise if 
the whole IF/ELSE would look like this
```java
runner.setProperty(PutTCP.CONNECTION_PER_FLOWFILE, connectionPerFlowFile ? 
"true" : "false");
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67732149
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles that 

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[joewitt]

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67732066
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles 

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67731299
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import javax.net.ssl.SSLContext;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ *
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter - A string to append to the end 
of each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles that 

[GitHub] nifi issue #543: NIFI-1834 Create PutTCP Processor

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/543
  
Looks good to me +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #513: PutHBaseJSON processor treats all values as Strings

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/513
  
@rtempleton just merged this work as part of PR 542, thanks for the 
contribution! 

can we close this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #542: NIFI-1895 for 0.x

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/542


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #547: NIFI-2060 - fixed unit tests to be compatible with 2...

[pvillard31]

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/547

NIFI-2060 - fixed unit tests to be compatible with 2.7.x

It’s a shame to limit what is tested here (in terms of event types) but 
it seems to be the only way to have both 2.6 and 2.7 lines working.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-2060

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/547.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #547


commit 41110598d8b32ce58b94338eee89044575c2095d
Author: Pierre Villard 
Date:   2016-06-20T17:15:09Z

NIFI-2060 - fixed unit tests to be compatible with 2.7.x




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #542: NIFI-1895 for 0.x

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/542
  
Specifically looking at Bryan's commit: I looked through the code and built 
locally (since travis failed for un-related issues) and it looks good. 

+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #546: NIFI-1877, NIFI-1306: Add fields to FlowFile for FIF...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/546

NIFI-1877, NIFI-1306: Add fields to FlowFile for FIFO Prioritizer, Ol…

…dest/Newest FlowFile first prioritizers to work property

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-1877

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/546.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #546


commit fce80761f886a53ff3ad5450c3524bf8510c9ea7
Author: Mark Payne 
Date:   2016-06-20T17:06:10Z

NIFI-1877, NIFI-1306: Add fields to FlowFile for FIFO Prioritizer, 
Oldest/Newest FlowFile first prioritizers to work property




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #398: NIFI-1751 Added proxy authentication in InvokeHttp processo...

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/398
  
Yep, not sure it's worth it. I think it is fine to let the PR as is.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: HDFS inotify commit to nifi-hdfs-processors seems to break MapR "profile" / Hadoop 2.7(?)

[Pierre Villard]

The way events are instantiated is not the same between 2.6.x and 2.7.x.
The easiest solution is to simplify unit tests to have tests both
compatible with 2.6 and 2.7 lines. I can submit a PR for that.
The other option would be to upgrade the version of Hadoop libraries used
in NiFi. However, Inotify mechanism is not considered production-ready at
the moment, so I guess an upgrade of version is not that necessary.

2016-06-20 10:35 GMT+02:00 Andre :

> Pierre,
>
> Yeah... the wonders of interoperation! :-)
>
> The classes came straight of of Hadoop 2.7 so compiling nifi with pure ASL
> Hadoop 2.7.0 triggers the same issue.
>
> Cheers
>
> On Mon, Jun 20, 2016 at 5:51 PM, Pierre Villard <
> pierre.villard...@gmail.com
> > wrote:
>
> > Thanks Andre for reporting this!
> > I'll try to have a look today but my MapR knowledge is limited...
> However,
> > since it seems to be a problem in test classes only, it should be
> possible
> > to find a workaround quite easily.
> >
> > Pierre
> >
> > 2016-06-18 17:43 GMT+02:00 Andre :
> >
> > > And if I my poor JAVA hasn't failed me again this seems to be the
> commit
> > > that introduced the upstream change
> > >
> > >
> > >
> >
> https://github.com/apache/hadoop/commit/6e13fc62e1f284f22fd0089f06ce281198bc7c2a
> > >
> > > https://issues.apache.org/jira/browse/HDFS-7513
> > >
> > > On Sun, Jun 19, 2016 at 1:36 AM, Andre  wrote:
> > >
> > > > Pierre,
> > > >
> > > > My build is currently hitting a wall when trying to
> > > > compile nifi-hdfs-processors.
> > > >
> > > > Seems the reason for this is the commit  2ce7857.
> > > >
> > > > Both
> > > >
> > > > $ mvn clean install -Dhadoop.version=2.7.0
> > > >
> > > > and
> > > >
> > > > $ mvn clean install -DskipTests -Pmapr
> -Dhadoop.version=2.7.0-mapr-1506
> > > >
> > > > result in:
> > > >
> > > > [ERROR] COMPILATION ERROR :
> > > > [INFO] -
> > > > [ERROR]
> > > >
> > >
> >
> /home/amiranda/development/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/inotify/TestGetHDFSEvents.java:[198,38]
> > > > error: incompatible types: String cannot be converted to Builder
> > > > [ERROR]
> > > >
> > >
> >
> /home/amiranda/development/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/inotify/TestGetHDFSEvents.java:[199,38]
> > > > error: incompatible types: String cannot be converted to Builder
> > > > [ERROR]
> > > >
> > >
> >
> /home/amiranda/development/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/inotify/TestGetHDFSEvents.java:[200,38]
> > > > error: incompatible types: String cannot be converted to Builder
> > > > [ERROR]
> > > >
> > >
> >
> /home/amiranda/development/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/inotify/util/EventTestUtils.java:[51,37]
> > > > error: incompatible types: String cannot be converted to Builder
> > > > [ERROR]
> > > >
> > >
> >
> /home/amiranda/development/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/inotify/util/EventTestUtils.java:[55,15]
> > > > error: constructor RenameEvent in class RenameEvent cannot be applied
> > to
> > > > given types;
> > > > [ERROR]   required: Builder
> > > >   found: String,String,long
> > > >   reason: actual and formal argument lists differ in length
> > > >
> > >
> >
> /home/amiranda/development/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/inotify/util/EventTestUtils.java:[75,15]
> > > > error: constructor UnlinkEvent in class UnlinkEvent cannot be applied
> > to
> > > > given types;
> > > > [INFO] 6 errors
> > > >
> > > >
> > > >
> > >
> >
>

[GitHub] nifi pull request #542: NIFI-1895 for 0.x

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/542#discussion_r67723205
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseJSON.java
 ---
@@ -89,6 +89,25 @@
 .defaultValue(COMPLEX_FIELD_TEXT.getValue())
 .build();
 
+protected static final String STRING_ENCODING_VALUE = "String";
+protected static final String BYTES_ENCODING_VALUE = "Bytes";
+
+protected static final AllowableValue FIELD_ENCODING_STRING = new 
AllowableValue(STRING_ENCODING_VALUE, STRING_ENCODING_VALUE,
+"Stores the value of each field as a UTF-8 String.");
--- End diff --

That works for me.

That's fair, if questions come in to the mailing list asking to clarify 
then we can address the description.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67722269
  
--- Diff: 
nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/put/AbstractPutEventProcessor.java
 ---
@@ -119,7 +119,24 @@
 .defaultValue("10 seconds")
 .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
 .build();
-
+public static final PropertyDescriptor OUTGOING_MESSAGE_DELIMITER = 
new PropertyDescriptor.Builder()
+.name("Outgoing Message Delimiter")
+.description("Specifies the delimiter to use when sending 
messages out over the same TCP stream. The delimiter is appended to each 
FlowFile message "
++ "that is transmitted over the stream so that the 
receiver can determine when one message ends and the next message begins. Users 
should "
++ "ensure that the FlowFile content does not contain 
the delimiter character to avoid errors. If it is not possible to define a 
delimiter "
++ "character that is not present in the FlowFile 
content then the user must use another processor to change the encoding of the 
data e.g. Base64 "
++ "encoding.")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+public static final PropertyDescriptor CONNECTION_PER_FLOWFILE = new 
PropertyDescriptor.Builder()
--- End diff --

While I'm not sure about use-cases I don't see a problem adding it as a 
property which defaults to false. I assume @aviewfromspace1 had a specific 
use-case in mind when adding the property and since it is optional I like 
keeping it.

Also PutTCP is marked as "TriggerWhenEmpty" so it will properly 
pruneIdleSenders just like PutSplunk.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

ControllerService Enabling and Processor onPropertyModified

[Michael D. Coon]

All,
   Before I get too deep in submitting Jira tickets, etc. I'm wondering if this 
is expected behavior. I'm using NiFi 0.6.1.

   I have a ControllerService that I reference as a service property on my 
Processor. The Processor, in turn, uses the ControllerService's internal 
configuration state to determine Processor output relationships. But, it 
appears that at NiFi startup, I'm given the ControllerService before it is 
actually enabled. When I try to invoke the  methods to get the 
ControllerService's state, it fails (because it's not enabled).
   Two problems I found:
1) Logging the invocation exception for calling a disabled ControllerService is 
being suppressed in this case, which caused this to take me a full day to track 
down2) Why would I ever be given a service to use without it being fully 
enabled?
I thought I would just block in my Processor's onPropertyModified method until 
the ControllerService was enabled; but it looks like the thread that's actually 
enabling the service is calling my Processor's onPropertyModified method and 
the ControllerService is never enabled until my Processor's onPropertyModified 
method is done.
Is this expected behavior? If so, can someone please explain the assumptions 
around sending non-enabled ControllerServices to my Processor?

Mike

[GitHub] nifi pull request #542: NIFI-1895 for 0.x

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/542#discussion_r67718748
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseJSON.java
 ---
@@ -89,6 +89,25 @@
 .defaultValue(COMPLEX_FIELD_TEXT.getValue())
 .build();
 
+protected static final String STRING_ENCODING_VALUE = "String";
+protected static final String BYTES_ENCODING_VALUE = "Bytes";
+
+protected static final AllowableValue FIELD_ENCODING_STRING = new 
AllowableValue(STRING_ENCODING_VALUE, STRING_ENCODING_VALUE,
+"Stores the value of each field as a UTF-8 String.");
--- End diff --

The existing PutHBase always produced UTF-8 so I think we are ok there.

I couldn't come up with great names for these choices so I would be open to 
something else if there was a better name, but I think if people read the 
descriptions of the values it will be clear.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #542: NIFI-1895 for 0.x

[rtempleton]

Github user rtempleton commented on the issue:

https://github.com/apache/nifi/pull/542
  
Good point, thanks @bbende, I hadn't thought to leave a backward 
compatibility feature in there.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #542: NIFI-1895 for 0.x

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/542#discussion_r67714535
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseJSON.java
 ---
@@ -89,6 +89,25 @@
 .defaultValue(COMPLEX_FIELD_TEXT.getValue())
 .build();
 
+protected static final String STRING_ENCODING_VALUE = "String";
+protected static final String BYTES_ENCODING_VALUE = "Bytes";
+
+protected static final AllowableValue FIELD_ENCODING_STRING = new 
AllowableValue(STRING_ENCODING_VALUE, STRING_ENCODING_VALUE,
+"Stores the value of each field as a UTF-8 String.");
--- End diff --

Secondly, will users configuring this processor know what a "String" is 
(aka. is it a concept that is used regularly when configuring HBase)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #542: NIFI-1895 for 0.x

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/542#discussion_r67713339
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseJSON.java
 ---
@@ -89,6 +89,25 @@
 .defaultValue(COMPLEX_FIELD_TEXT.getValue())
 .build();
 
+protected static final String STRING_ENCODING_VALUE = "String";
+protected static final String BYTES_ENCODING_VALUE = "Bytes";
+
+protected static final AllowableValue FIELD_ENCODING_STRING = new 
AllowableValue(STRING_ENCODING_VALUE, STRING_ENCODING_VALUE,
+"Stores the value of each field as a UTF-8 String.");
--- End diff --

Should the charset be selectable or will it always be UTF-8?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Closing in on the Apache NiFi 0.7.0 Release

[Bryan Bende]

I believe NIFI-1895 should be ready to go for 0.7.0 as well...

https://github.com/apache/nifi/pull/542

On Mon, Jun 20, 2016 at 11:39 AM, Joe Percivall <
joeperciv...@yahoo.com.invalid> wrote:

> Team,
>
> We are doing a great job finishing up tickets. Six more were closed over
> the weekend and most of the others made progress as well. We only have
> three more that are tagged as 0.7.0 in Jira. There are a couple others
> which are unversioned that are close but haven't been identified as
> important enough to hold up the release for so I will not reflect them here.
>
> The current status for those left tagged as 0.7.0:
>
> - "Processors could be started before the Controller Services that they
> depend on" NIFI-2032[1] [status] Oleg Zhurakousky's PR was looked at by
> Andy LoPresto but not clear if more action is needed.
>
> - "Create a processor to extract WAV file characteristics" NIFI-615[2]
> [status] Waiting for Joe Skora to rebase and put changes into a PR (which
> he said he would do today) and for final review from Joe Percivall (me)
> once the changes are made.
>
> - "Create PutTCP Processor" NIFI-1834[3] [status] I took Matt Brown's
> original commit and applied my comments in another commit. Bryan Bende
> reviewed and I am addressing now.
>
> Also NIFI-1974 and NIFI-1850 are marked as 0.7.0 and "Patch Available" in
> Jira but they are already merged into 0.x and work is being done to get it
> into master. I will not reflect them here.
>
> [1] https://issues.apache.org/jira/browse/NIFI-2032
> [2] https://issues.apache.org/jira/browse/NIFI-615
>
> [3] https://issues.apache.org/jira/browse/NIFI-1834
>
>
> Joe
> - - - - - -
> Joseph Percivalllinkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
>
>
>
> On Friday, June 17, 2016 2:23 PM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
>
>
>
> 2009 is all done (including 0.x)
>
> Sent from my iPhone
>
>
> > On Jun 17, 2016, at 14:16, Joe Percivall 
> wrote:
> >
> > Team,
> >
> > We are continuing to have good progress towards the 0.7.0 release.
> Tagged as "0.7.0" in Jira and still pending, we currently have 9 tickets
> that are all patch available.
> >
> > As a note, there seem to be a number of unversioned "patch available"
> tickets in Jira. I am going to take a look at them to see if any can easily
> make it into 0.7.0 (or are critical) but unless if they are critical I am
> not going to hold up the release for them.
> >
> > The current status for those tagged as 0.7.0:
> >
> > - "Processors could be started before the Controller Services that they
> depend on" NIFI-2032[1] [status] Critical bug that was added two days ago.
> Oleg Zhurakousky posted a PR and needs review.
> >
> >
> > - "Allow user to specify file filter regex when unpacking zip/tar
> archives" NIFI-1568[2] [status] Matt Burgess finished reviewing and is
> waiting for Ricky Saltzer to address the final comment.
> >
> > - "Create PutSlack processor" NIFI-1578 [3] [status] Matt Burgess found
> one final issue before it is committed. Waiting for Adam Lamar to address.
> >
> > -  "Allow empty Content-Type in InvokeHTTP processor" NIFI-1620[4]
> [status] Pierre Villard addressed Adam Taft's comments is waiting for a
> final review.
> >
> > - "Support Custom Properties in Expression Language" NIFI-1974[5]
> [status] No Change, waiting for Mark Payne to give a final review.
> >
> > - "StandardProcessNode and AbstractConfiguredComponent duplicate
> instance variable "annotationData"" NIFI-2009[6] [status] Was merged into
> master but needs to fix merge conflicts to be merged into 0.x
> > - "Create a processor to extract WAV file characteristics" NIFI-615[7]
> [status] Joe Skora posted a new branch addressing the comments but needs to
> rebase and update PR. Branch needs final review from Joe Percivall (me)
> >
> > - "Add SNMP processors" NIFI-1537[8] [status] PR is rebased and Oleg
> Zhurakousky will be reviewing
> >
> > - "Create FlowDebugger processor" NIFI-1829[9] [status] Joe Skora added
> another commit and rebased. Tony won't be able to finalize review until the
> weekend, Michael Moser volunteered to finish
> >
> >
> > Also NiFi-1850 is marked as 0.7.0 and "Patch Available" in Jira but it
> was already was merged into 0.x and work is being done to get it into
> master. I will not reflect it here.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-2032
> > [2] https://issues.apache.org/jira/browse/NIFI-1568
> > [3] https://issues.apache.org/jira/browse/NIFI-1578
> > [4] https://issues.apache.org/jira/browse/NIFI-1620
> > [5] https://issues.apache.org/jira/browse/NIFI-1974
> > [6] https://issues.apache.org/jira/browse/NIFI-2009
> > [7] https://issues.apache.org/jira/browse/NIFI-615
> > [8] https://issues.apache.org/jira/browse/NIFI-1537
> > [9] https://issues.apache.org/jira/browse/NIFI-1829
> >
> > Joe
> > - - - - - -
> > Joseph Percivalllinkedin.com/in/Percivall
> > e: joeperciv...@yahoo.com
> >
> >
> >
> >
> > On Wednesday, June 15, 

Re: [DISCUSS] Closing in on the Apache NiFi 0.7.0 Release

[Joe Percivall]

Team,

We are doing a great job finishing up tickets. Six more were closed over the 
weekend and most of the others made progress as well. We only have three more 
that are tagged as 0.7.0 in Jira. There are a couple others which are 
unversioned that are close but haven't been identified as important enough to 
hold up the release for so I will not reflect them here.

The current status for those left tagged as 0.7.0:

- "Processors could be started before the Controller Services that they depend 
on" NIFI-2032[1] [status] Oleg Zhurakousky's PR was looked at by Andy LoPresto 
but not clear if more action is needed. 

- "Create a processor to extract WAV file characteristics" NIFI-615[2] [status] 
Waiting for Joe Skora to rebase and put changes into a PR (which he said he 
would do today) and for final review from Joe Percivall (me) once the changes 
are made.

- "Create PutTCP Processor" NIFI-1834[3] [status] I took Matt Brown's original 
commit and applied my comments in another commit. Bryan Bende reviewed and I am 
addressing now.

Also NIFI-1974 and NIFI-1850 are marked as 0.7.0 and "Patch Available" in Jira 
but they are already merged into 0.x and work is being done to get it into 
master. I will not reflect them here.

[1] https://issues.apache.org/jira/browse/NIFI-2032
[2] https://issues.apache.org/jira/browse/NIFI-615

[3] https://issues.apache.org/jira/browse/NIFI-1834


Joe
- - - - - - 
Joseph Percivalllinkedin.com/in/Percivall
e: joeperciv...@yahoo.com



On Friday, June 17, 2016 2:23 PM, Oleg Zhurakousky 
 wrote:



2009 is all done (including 0.x)

Sent from my iPhone


> On Jun 17, 2016, at 14:16, Joe Percivall  
> wrote:
> 
> Team,
> 
> We are continuing to have good progress towards the 0.7.0 release. Tagged as 
> "0.7.0" in Jira and still pending, we currently have 9 tickets that are all 
> patch available. 
> 
> As a note, there seem to be a number of unversioned "patch available" tickets 
> in Jira. I am going to take a look at them to see if any can easily make it 
> into 0.7.0 (or are critical) but unless if they are critical I am not going 
> to hold up the release for them. 
> 
> The current status for those tagged as 0.7.0:
> 
> - "Processors could be started before the Controller Services that they 
> depend on" NIFI-2032[1] [status] Critical bug that was added two days ago. 
> Oleg Zhurakousky posted a PR and needs review.
> 
> 
> - "Allow user to specify file filter regex when unpacking zip/tar archives" 
> NIFI-1568[2] [status] Matt Burgess finished reviewing and is waiting for 
> Ricky Saltzer to address the final comment.
> 
> - "Create PutSlack processor" NIFI-1578 [3] [status] Matt Burgess found one 
> final issue before it is committed. Waiting for Adam Lamar to address.
> 
> -  "Allow empty Content-Type in InvokeHTTP processor" NIFI-1620[4] [status] 
> Pierre Villard addressed Adam Taft's comments is waiting for a final review.
> 
> - "Support Custom Properties in Expression Language" NIFI-1974[5] [status] No 
> Change, waiting for Mark Payne to give a final review.
> 
> - "StandardProcessNode and AbstractConfiguredComponent duplicate instance 
> variable "annotationData"" NIFI-2009[6] [status] Was merged into master but 
> needs to fix merge conflicts to be merged into 0.x
> - "Create a processor to extract WAV file characteristics" NIFI-615[7] 
> [status] Joe Skora posted a new branch addressing the comments but needs to 
> rebase and update PR. Branch needs final review from Joe Percivall (me)
> 
> - "Add SNMP processors" NIFI-1537[8] [status] PR is rebased and Oleg 
> Zhurakousky will be reviewing
> 
> - "Create FlowDebugger processor" NIFI-1829[9] [status] Joe Skora added 
> another commit and rebased. Tony won't be able to finalize review until the 
> weekend, Michael Moser volunteered to finish
> 
> 
> Also NiFi-1850 is marked as 0.7.0 and "Patch Available" in Jira but it was 
> already was merged into 0.x and work is being done to get it into master. I 
> will not reflect it here.
> 
> [1] https://issues.apache.org/jira/browse/NIFI-2032
> [2] https://issues.apache.org/jira/browse/NIFI-1568
> [3] https://issues.apache.org/jira/browse/NIFI-1578
> [4] https://issues.apache.org/jira/browse/NIFI-1620
> [5] https://issues.apache.org/jira/browse/NIFI-1974
> [6] https://issues.apache.org/jira/browse/NIFI-2009
> [7] https://issues.apache.org/jira/browse/NIFI-615
> [8] https://issues.apache.org/jira/browse/NIFI-1537
> [9] https://issues.apache.org/jira/browse/NIFI-1829
> 
> Joe
> - - - - - - 
> Joseph Percivalllinkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
> 
> 
> 
> 
> On Wednesday, June 15, 2016 10:56 AM, Joe Percivall 
>  wrote:
> Team,
> 
> There was a lot of great progress yesterday, we closed or pushed 6 tickets. 
> Also two tickets were added that are either critical or finishing shortly. 
> The status of the remaining 12 tickets are below:
> 
> - "Corrupted flow 

[GitHub] nifi issue #398: NIFI-1751 Added proxy authentication in InvokeHttp processo...

[apiri]

Github user apiri commented on the issue:

https://github.com/apache/nifi/pull/398
  
Actually, I think the Response#priorResponse probably doesn't apply to our 
case after thinking about it a bit more.  The key way to avoid the issues seen 
is a flag/counter within that authenticator to track if an attempt to 
authenticate has been made and then throw an exception in that case.  I think 
the handling is fine now, but we can bypass the HTTP client's internal retries.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #525: NIFI-1976 - Windows Event Log native processor

[brosander]

Github user brosander commented on the issue:

https://github.com/apache/nifi/pull/525
  
@JPercivall I believe I have incorporated all of the suggested changes 
(except for the XPath validator because it didn't like some of the provided 
examples in the api doc).  Mind taking another look?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #248: NIFI-1568: Add Filter Capability to UnpackContent

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/248


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #362: NIFI-1769: added support for SSE-KMS and signature s3v4 aut...

[jvwing]

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/362
  
@miquillo, thanks for putting together this PR, Server-Side Encryption with 
customer keys is a great feature for NiFi to have.  I'm doing some review, and 
considering the following topics:

* **NiFi's AWS SDK Version** - Since you submitted this PR, Amazon appears 
to have made signature version 4 the default for S3 in their Java SDK v1.11.0, 
May 13, 2016.  NiFi's current SDK version is 1.10.32 from Nov 3, 2015.  
Upgrading the SDK would apply much broader than this feature, but maybe not out 
of line for the v1.0 release.  If we chose not to upgrade now, we might 
consider that in a future upgrade the signature version defaults will change.
* **Controlling Signature Versions** - Regardless of the SDK default, you 
are absolutely right to make it optional.  I'm a bit baffled by why Amazon uses 
a System property to control this setting rather than a client or request 
property, and a bit concerned about multiple processors fighting over the 
signature version.  I agree that `System.setProperty(...)` is the [documented 
method of setting the signature 
version](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html).  But 
I don't like it, I would prefer to set this more concisely than a global 
setting.  Are you familiar with 
[`ClientConfiguration::setSignerOverride()`](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html#setSignerOverride(java.lang.String))?
  It appears to allow this, if not so well documented.
* **SSE KMS for FetchS3Object** - We might also want to apply this feature 
to the FetchS3Object processor, or at least to allow for that to happen in the 
future.  Have you considered moving some of the KMS logic to the 
AbstractS3Processor class? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67707758
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.stream.io.StreamUtils;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ * 
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter/b> - A string to append to the end of 
each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles that failed to be 
sent.
+ * success - Where to route FlowFiles after they were 
successfully sent to the TCP server.
+ * 
+ * 
+ *
+ */

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67707310
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.stream.io.StreamUtils;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ * 
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter/b> - A string to append to the end of 
each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles that failed to be 
sent.
+ * success - Where to route FlowFiles after they were 
successfully sent to the TCP server.
+ * 
+ * 
+ *
+ */

[GitHub] nifi issue #398: NIFI-1751 Added proxy authentication in InvokeHttp processo...

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/398
  
Hey @apiri,

Many thanks for the review!

Regarding HTTPS, at the time I submitted the PR I was testing it with 
CCProxy (on Windows computer). I can't test it again with this setup before the 
WE but if you didn't see the issue, it sounds good to me :)

I'll try to setup Squid on my Mac and add a control using the 
``Response.priorResponse()`` but, right now, not sure to see how this should be 
done (get the previous response and compare it with the new one and stop if 
looping ?). I'll keep you posted unless you want to do it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67700294
  
--- Diff: 
nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/put/AbstractPutEventProcessor.java
 ---
@@ -126,8 +127,9 @@
 + "ensure that the FlowFile content does not contain 
the delimiter character to avoid errors. If it is not possible to define a 
delimiter "
 + "character that is not present in the FlowFile 
content then the user must use another processor to change the encoding of the 
data e.g. Base64 "
 + "encoding.")
-.required(false)
+.required(true)
 .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.defaultValue("\n")
--- End diff --

I think this should be "\\n" so it shows up in the property of the 
processor, looks like the code already replaces "\\n" to "\n" before using the 
value 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67698814
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.stream.io.StreamUtils;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ * 
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter/b> - A string to append to the end of 
each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles that failed to be 
sent.
+ * success - Where to route FlowFiles after they were 
successfully sent to the TCP server.
+ * 
+ * 
+ *
+ */

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67698292
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.stream.io.StreamUtils;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ * 
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter/b> - A string to append to the end of 
each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles that failed to be 
sent.
+ * success - Where to route FlowFiles after they were 
successfully sent to the TCP server.
+ * 
+ * 
+ *
+ */

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67696982
  
--- Diff: 
nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/put/AbstractPutEventProcessor.java
 ---
@@ -119,7 +119,24 @@
 .defaultValue("10 seconds")
 .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
 .build();
-
+public static final PropertyDescriptor OUTGOING_MESSAGE_DELIMITER = 
new PropertyDescriptor.Builder()
+.name("Outgoing Message Delimiter")
+.description("Specifies the delimiter to use when sending 
messages out over the same TCP stream. The delimiter is appended to each 
FlowFile message "
++ "that is transmitted over the stream so that the 
receiver can determine when one message ends and the next message begins. Users 
should "
++ "ensure that the FlowFile content does not contain 
the delimiter character to avoid errors. If it is not possible to define a 
delimiter "
++ "character that is not present in the FlowFile 
content then the user must use another processor to change the encoding of the 
data e.g. Base64 "
++ "encoding.")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+public static final PropertyDescriptor CONNECTION_PER_FLOWFILE = new 
PropertyDescriptor.Builder()
--- End diff --

I'm wondering if we need this property... the pattern in AbstractPutEvent 
is to have a blocking queue of connections, up to the # of concurrent tasks for 
the processor. At the beginning of onTrigger it tries to acquire a connection, 
creates a new one if none available, and at the end returns to the queue unless 
the queue is full then it closes. There is also the concept of idle connections 
which get closed if they haven't been used in a certain amount of time. So 
unless there is a specific use case where we would want a new connection every 
single time, then I would opt for the above behavior.

The only thing we might have to consider is that PutSplunk called 
pruneIdleSenders when flowFile == null, which would happen more frequently 
because it had @TriggerWhenEmpty. With this processor maybe we should be 
calling prune based on some other condition?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #397: NIFI-1815

[olegz]

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/397
  
back to reviewing this. It has some merge conflicts but i'll fox them as I 
go along


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/543#discussion_r67697377
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java
 ---
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.put.AbstractPutEventProcessor;
+import org.apache.nifi.processor.util.put.sender.ChannelSender;
+import org.apache.nifi.stream.io.StreamUtils;
+
+/**
+ * 
+ * The PutTCP processor receives a FlowFile and transmits the FlowFile 
content over a TCP connection to the configured TCP server. By default, the 
FlowFiles are transmitted over the same TCP
+ * connection (or pool of TCP connections if multiple input threads are 
configured). To assist the TCP server with determining message boundaries, an 
optional "Outgoing Message Delimiter" string can
+ * be configured which is appended to the end of each FlowFiles content 
when it is transmitted over the TCP connection. An optional "Connection Per 
FlowFile" parameter can be specified to change the
+ * behaviour so that each FlowFiles content is transmitted over a single 
TCP connection which is opened when the FlowFile is received and closed after 
the FlowFile has been sent. This option should
+ * only be used for low message volume scenarios, otherwise the platform 
may run out of TCP sockets.
+ * 
+ *
+ * 
+ * This processor has the following required properties:
+ * 
+ * Hostname - The IP address or host name of the destination 
TCP server.
+ * Port - The TCP port of the destination TCP server.
+ * 
+ * 
+ * 
+ * 
+ * This processor has the following optional properties:
+ * 
+ * Connection Per FlowFile - Specifies that each FlowFiles 
content will be transmitted on a separate TCP connection.
+ * Idle Connection Expiration - The time threshold after which 
a TCP sender is deemed eligible for pruning - the associated TCP connection 
will be closed after this timeout.
+ * Max Size of Socket Send Buffer - The maximum size of the 
socket send buffer that should be used. This is a suggestion to the Operating 
System to indicate how big the socket buffer should
+ * be. If this value is set too low, the buffer may fill up before the 
data can be read, and incoming data will be dropped.
+ * Outgoing Message Delimiter/b> - A string to append to the end of 
each FlowFiles content to indicate the end of the message to the TCP 
server.
+ * Timeout - The timeout period for determining an error has 
occurred whilst connecting or sending data.
+ * 
+ * 
+ *
+ * 
+ * The following relationships are required:
+ * 
+ * failure - Where to route FlowFiles that failed to be 
sent.
+ * success - Where to route FlowFiles after they were 
successfully sent to the TCP server.
+ * 
+ * 
+ *
+ */

[GitHub] nifi pull request #545: NIFI-2009 port for 0.x

[olegz]

Github user olegz closed the pull request at:

https://github.com/apache/nifi/pull/545


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #545: NIFI-2009 port for 0.x

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/545
  
Thanks @olegz! I just merged it into 0.x, could you close the PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: HDFS inotify commit to nifi-hdfs-processors seems to break MapR "profile" / Hadoop 2.7(?)

[Pierre Villard]

Thanks Andre for reporting this!
I'll try to have a look today but my MapR knowledge is limited... However,
since it seems to be a problem in test classes only, it should be possible
to find a workaround quite easily.

Pierre

2016-06-18 17:43 GMT+02:00 Andre :

> And if I my poor JAVA hasn't failed me again this seems to be the commit
> that introduced the upstream change
>
>
> https://github.com/apache/hadoop/commit/6e13fc62e1f284f22fd0089f06ce281198bc7c2a
>
> https://issues.apache.org/jira/browse/HDFS-7513
>
> On Sun, Jun 19, 2016 at 1:36 AM, Andre  wrote:
>
> > Pierre,
> >
> > My build is currently hitting a wall when trying to
> > compile nifi-hdfs-processors.
> >
> > Seems the reason for this is the commit  2ce7857.
> >
> > Both
> >
> > $ mvn clean install -Dhadoop.version=2.7.0
> >
> > and
> >
> > $ mvn clean install -DskipTests -Pmapr -Dhadoop.version=2.7.0-mapr-1506
> >
> > result in:
> >
> > [ERROR] COMPILATION ERROR :
> > [INFO] -
> > [ERROR]
> >
> /home/amiranda/development/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/inotify/TestGetHDFSEvents.java:[198,38]
> > error: incompatible types: String cannot be converted to Builder
> > [ERROR]
> >
> /home/amiranda/development/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/inotify/TestGetHDFSEvents.java:[199,38]
> > error: incompatible types: String cannot be converted to Builder
> > [ERROR]
> >
> /home/amiranda/development/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/inotify/TestGetHDFSEvents.java:[200,38]
> > error: incompatible types: String cannot be converted to Builder
> > [ERROR]
> >
> /home/amiranda/development/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/inotify/util/EventTestUtils.java:[51,37]
> > error: incompatible types: String cannot be converted to Builder
> > [ERROR]
> >
> /home/amiranda/development/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/inotify/util/EventTestUtils.java:[55,15]
> > error: constructor RenameEvent in class RenameEvent cannot be applied to
> > given types;
> > [ERROR]   required: Builder
> >   found: String,String,long
> >   reason: actual and formal argument lists differ in length
> >
> /home/amiranda/development/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/inotify/util/EventTestUtils.java:[75,15]
> > error: constructor UnlinkEvent in class UnlinkEvent cannot be applied to
> > given types;
> > [INFO] 6 errors
> >
> >
> >
>