[jira] [Commented] (NIFI-4105) support the specified Maximum value column and CSV Stream for Cassandra

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069482#comment-16069482
 ] 

ASF GitHub Bot commented on NIFI-4105:
--

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124964375
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/test/java/org/apache/nifi/processors/cassandra/QueryCassandraTest.java
 ---
@@ -361,6 +362,14 @@ public void testConvertToJSONStream() throws Exception 
{
 assertEquals(2, numberOfRows);
 }
 
+@Test
+public void testConvertToCSVStream() throws Exception {
--- End diff --

Sure! I forgot to push. 


> support the specified Maximum value column and CSV Stream for Cassandra
> ---
>
> Key: NIFI-4105
> URL: https://issues.apache.org/jira/browse/NIFI-4105
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.3.0
>Reporter: Yoonwon Ko
>
> I'm trying to find a CassandraProcessor to fetch rows whose values in the 
> specified Maximum Value columns are larger than the previously-seen maximum 
> like QueryDatabaseTable.
> But I found only QueryCassandra. It just executes same CQL everytime without 
> keeping maximum value.
> and I think we also need convertToCsvStream option.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1937: NIFI-4105 support the specified Maximum value colum...

[ggthename]

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124964375
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/test/java/org/apache/nifi/processors/cassandra/QueryCassandraTest.java
 ---
@@ -361,6 +362,14 @@ public void testConvertToJSONStream() throws Exception 
{
 assertEquals(2, numberOfRows);
 }
 
+@Test
+public void testConvertToCSVStream() throws Exception {
--- End diff --

Sure! I forgot to push. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4105) support the specified Maximum value column and CSV Stream for Cassandra

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069479#comment-16069479
 ] 

ASF GitHub Bot commented on NIFI-4105:
--

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124964237
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
 ---
@@ -75,19 +81,64 @@
 @Tags({"cassandra", "cql", "select"})
 @EventDriven
 @InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
-@CapabilityDescription("Execute provided Cassandra Query Language (CQL) 
select query on a Cassandra 1.x, 2.x, or 3.0.x cluster. Query result "
-+ "may be converted to Avro or JSON format. Streaming is used so 
arbitrarily large result sets are supported. This processor can be "
+@CapabilityDescription("Executes provided Cassandra Query Language (CQL) 
select query on a Cassandra to fetch all rows whose values"
++ "in the specified Maximum Value column(s) are larger than the 
previously-seen maxima.Query result"
++ "may be converted to Avro, JSON or CSV format. Streaming is used 
so arbitrarily large result sets are supported. This processor can be "
 + "scheduled to run on a timer, or cron expression, using the 
standard scheduling methods, or it can be triggered by an incoming FlowFile. "
 + "If it is triggered by an incoming FlowFile, then attributes of 
that FlowFile will be available when evaluating the "
 + "select query. FlowFile attribute 'executecql.row.count' 
indicates how many rows were selected.")
+@Stateful(scopes = Scope.CLUSTER, description = "After performing query, 
the maximum value of the specified column is stored, "
++ "fetch all rows whose values in the specified Maximum Value 
column(s) are larger than the previously-seen maximum"
++ "State is stored across the cluster so that the next time this 
Processor can be run with min and max values")
 @WritesAttributes({@WritesAttribute(attribute = "executecql.row.count", 
description = "The number of rows returned by the CQL query")})
 public class QueryCassandra extends AbstractCassandraProcessor {
 
+public static final String CSV_FORMAT = "CSV";
 public static final String AVRO_FORMAT = "Avro";
 public static final String JSON_FORMAT = "JSON";
 
+public static final String CASSANDRA_WATERMARK_MIN_VALUE_ID = 
"CASSANDRA_WATERMARK_MIN_VALUE_ID";
+public static final String CASSANDRA_WATERMARK_MAX_VALUE_ID = 
"CASSANDRA_WATERMARK_MAX_VALUE_ID";
+
 public static final String RESULT_ROW_COUNT = "executecql.row.count";
 
+public static final PropertyDescriptor INIT_WATERMARK = new 
PropertyDescriptor.Builder().name("Initial Watermark Value")
+.description("Use it only once.")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor BACKOFF_PERIOD = new 
PropertyDescriptor.Builder()
+.name("Backoff Period")
+.description("Only records older than the backoff period will 
be eligible for pickup. This can be used in the ILM use case to define a 
retention period.")
+.defaultValue("10 seconds")
+.required(true)
+.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+.sensitive(false)
+.build();
+
+public static final PropertyDescriptor OVERLAP_TIME = new 
PropertyDescriptor.Builder()
+.name("Overlap Period")
+.description("Amount of time to overlap into the last load 
date to ensure long running transactions missed by previous load weren't 
missed. Recommended: >0s")
+.required(true)
+.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+.defaultValue("0 seconds")
+.build();
+
+public static final PropertyDescriptor DATE_FIELD = new 
PropertyDescriptor.Builder()
--- End diff --

I think it's a good idea, but a little bit complex. 
because we usually want to concentrate the time of record when we use the 
incremental fetch feature. ( To avoid omissions )

So there are the concepts of BACKOFF_PERIOD and OVERLAP_TIME.
It is difficult to apply the other type column.

In my opinion, a date column is most appropriate for  the incremental fetch 
feature.


> support the specified Maximum value column and CSV Stream for Cassandra
> ---
>
> Key: NIFI-4105
> URL: 

[GitHub] nifi pull request #1937: NIFI-4105 support the specified Maximum value colum...

[ggthename]

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124964237
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
 ---
@@ -75,19 +81,64 @@
 @Tags({"cassandra", "cql", "select"})
 @EventDriven
 @InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
-@CapabilityDescription("Execute provided Cassandra Query Language (CQL) 
select query on a Cassandra 1.x, 2.x, or 3.0.x cluster. Query result "
-+ "may be converted to Avro or JSON format. Streaming is used so 
arbitrarily large result sets are supported. This processor can be "
+@CapabilityDescription("Executes provided Cassandra Query Language (CQL) 
select query on a Cassandra to fetch all rows whose values"
++ "in the specified Maximum Value column(s) are larger than the 
previously-seen maxima.Query result"
++ "may be converted to Avro, JSON or CSV format. Streaming is used 
so arbitrarily large result sets are supported. This processor can be "
 + "scheduled to run on a timer, or cron expression, using the 
standard scheduling methods, or it can be triggered by an incoming FlowFile. "
 + "If it is triggered by an incoming FlowFile, then attributes of 
that FlowFile will be available when evaluating the "
 + "select query. FlowFile attribute 'executecql.row.count' 
indicates how many rows were selected.")
+@Stateful(scopes = Scope.CLUSTER, description = "After performing query, 
the maximum value of the specified column is stored, "
++ "fetch all rows whose values in the specified Maximum Value 
column(s) are larger than the previously-seen maximum"
++ "State is stored across the cluster so that the next time this 
Processor can be run with min and max values")
 @WritesAttributes({@WritesAttribute(attribute = "executecql.row.count", 
description = "The number of rows returned by the CQL query")})
 public class QueryCassandra extends AbstractCassandraProcessor {
 
+public static final String CSV_FORMAT = "CSV";
 public static final String AVRO_FORMAT = "Avro";
 public static final String JSON_FORMAT = "JSON";
 
+public static final String CASSANDRA_WATERMARK_MIN_VALUE_ID = 
"CASSANDRA_WATERMARK_MIN_VALUE_ID";
+public static final String CASSANDRA_WATERMARK_MAX_VALUE_ID = 
"CASSANDRA_WATERMARK_MAX_VALUE_ID";
+
 public static final String RESULT_ROW_COUNT = "executecql.row.count";
 
+public static final PropertyDescriptor INIT_WATERMARK = new 
PropertyDescriptor.Builder().name("Initial Watermark Value")
+.description("Use it only once.")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor BACKOFF_PERIOD = new 
PropertyDescriptor.Builder()
+.name("Backoff Period")
+.description("Only records older than the backoff period will 
be eligible for pickup. This can be used in the ILM use case to define a 
retention period.")
+.defaultValue("10 seconds")
+.required(true)
+.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+.sensitive(false)
+.build();
+
+public static final PropertyDescriptor OVERLAP_TIME = new 
PropertyDescriptor.Builder()
+.name("Overlap Period")
+.description("Amount of time to overlap into the last load 
date to ensure long running transactions missed by previous load weren't 
missed. Recommended: >0s")
+.required(true)
+.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+.defaultValue("0 seconds")
+.build();
+
+public static final PropertyDescriptor DATE_FIELD = new 
PropertyDescriptor.Builder()
--- End diff --

I think it's a good idea, but a little bit complex. 
because we usually want to concentrate the time of record when we use the 
incremental fetch feature. ( To avoid omissions )

So there are the concepts of BACKOFF_PERIOD and OVERLAP_TIME.
It is difficult to apply the other type column.

In my opinion, a date column is most appropriate for  the incremental fetch 
feature.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4105) support the specified Maximum value column and CSV Stream for Cassandra

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069467#comment-16069467
 ] 

ASF GitHub Bot commented on NIFI-4105:
--

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124962998
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/test/java/org/apache/nifi/processors/cassandra/QueryCassandraTest.java
 ---
@@ -361,6 +362,14 @@ public void testConvertToJSONStream() throws Exception 
{
 assertEquals(2, numberOfRows);
 }
 
+@Test
+public void testConvertToCSVStream() throws Exception {
+ResultSet rs = CassandraQueryTestUtil.createMockResultSet();
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+long numberOfRows = QueryCassandra.convertToCsvStream(rs, baos, 
StandardCharsets.UTF_8, 0, null);
+assertEquals(2, numberOfRows);
--- End diff --

I'll add a test case which is called testProcessorCsvOutput.


> support the specified Maximum value column and CSV Stream for Cassandra
> ---
>
> Key: NIFI-4105
> URL: https://issues.apache.org/jira/browse/NIFI-4105
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.3.0
>Reporter: Yoonwon Ko
>
> I'm trying to find a CassandraProcessor to fetch rows whose values in the 
> specified Maximum Value columns are larger than the previously-seen maximum 
> like QueryDatabaseTable.
> But I found only QueryCassandra. It just executes same CQL everytime without 
> keeping maximum value.
> and I think we also need convertToCsvStream option.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1937: NIFI-4105 support the specified Maximum value colum...

[ggthename]

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124962998
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/test/java/org/apache/nifi/processors/cassandra/QueryCassandraTest.java
 ---
@@ -361,6 +362,14 @@ public void testConvertToJSONStream() throws Exception 
{
 assertEquals(2, numberOfRows);
 }
 
+@Test
+public void testConvertToCSVStream() throws Exception {
+ResultSet rs = CassandraQueryTestUtil.createMockResultSet();
+ByteArrayOutputStream baos = new ByteArrayOutputStream();
+long numberOfRows = QueryCassandra.convertToCsvStream(rs, baos, 
StandardCharsets.UTF_8, 0, null);
+assertEquals(2, numberOfRows);
--- End diff --

I'll add a test case which is called testProcessorCsvOutput.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4105) support the specified Maximum value column and CSV Stream for Cassandra

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069449#comment-16069449
 ] 

ASF GitHub Bot commented on NIFI-4105:
--

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124961912
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
 ---
@@ -485,6 +668,68 @@ public static long convertToJsonStream(final ResultSet 
rs, final OutputStream ou
 }
 }
 
+/**
+ * Converts a result set into an CSV record and writes it to the given 
stream using the specified character set.
+ *
+ * @param rsThe result set to convert
+ * @param outStream The stream to which the CSV record will be written
+ * @param timeout   The max number of timeUnits to wait for a result 
set fetch to complete
+ * @param timeUnit  The unit of time (SECONDS, e.g.) associated with 
the timeout amount
+ * @return The number of rows from the result set written to the stream
+ * @throws IOException  If the CSV record cannot be written
+ * @throws InterruptedException If a result set fetch is interrupted
+ * @throws TimeoutException If a result set fetch has taken longer 
than the specified timeout
+ * @throws ExecutionException   If any error occurs during the result 
set fetch
+ */
+public static long convertToCsvStream(final ResultSet rs, final 
OutputStream outStream, Charset charset,
--- End diff --

I think it should be optional. Because other processors have to connect 
with this processor using flowfile.
So I'll adding a boolean value.


> support the specified Maximum value column and CSV Stream for Cassandra
> ---
>
> Key: NIFI-4105
> URL: https://issues.apache.org/jira/browse/NIFI-4105
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.3.0
>Reporter: Yoonwon Ko
>
> I'm trying to find a CassandraProcessor to fetch rows whose values in the 
> specified Maximum Value columns are larger than the previously-seen maximum 
> like QueryDatabaseTable.
> But I found only QueryCassandra. It just executes same CQL everytime without 
> keeping maximum value.
> and I think we also need convertToCsvStream option.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1937: NIFI-4105 support the specified Maximum value colum...

[ggthename]

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124961912
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
 ---
@@ -485,6 +668,68 @@ public static long convertToJsonStream(final ResultSet 
rs, final OutputStream ou
 }
 }
 
+/**
+ * Converts a result set into an CSV record and writes it to the given 
stream using the specified character set.
+ *
+ * @param rsThe result set to convert
+ * @param outStream The stream to which the CSV record will be written
+ * @param timeout   The max number of timeUnits to wait for a result 
set fetch to complete
+ * @param timeUnit  The unit of time (SECONDS, e.g.) associated with 
the timeout amount
+ * @return The number of rows from the result set written to the stream
+ * @throws IOException  If the CSV record cannot be written
+ * @throws InterruptedException If a result set fetch is interrupted
+ * @throws TimeoutException If a result set fetch has taken longer 
than the specified timeout
+ * @throws ExecutionException   If any error occurs during the result 
set fetch
+ */
+public static long convertToCsvStream(final ResultSet rs, final 
OutputStream outStream, Charset charset,
--- End diff --

I think it should be optional. Because other processors have to connect 
with this processor using flowfile.
So I'll adding a boolean value.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-4037) Create gRPC client processor

[Tony Kurc (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tony Kurc updated NIFI-4037:

Fix Version/s: 1.4.0

> Create gRPC client processor 
> -
>
> Key: NIFI-4037
> URL: https://issues.apache.org/jira/browse/NIFI-4037
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
> Fix For: 1.4.0
>
> Attachments: nifi_invoke_grpc.png
>
>
> Create a simple gRPC [1] client processor similar to InvokeHTTP that would 
> allow  the sharing of flow files to a configured remote gRPC server.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-4037) Create gRPC client processor

[Tony Kurc (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tony Kurc resolved NIFI-4037.
-
Resolution: Done

> Create gRPC client processor 
> -
>
> Key: NIFI-4037
> URL: https://issues.apache.org/jira/browse/NIFI-4037
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
> Fix For: 1.4.0
>
> Attachments: nifi_invoke_grpc.png
>
>
> Create a simple gRPC [1] client processor similar to InvokeHTTP that would 
> allow  the sharing of flow files to a configured remote gRPC server.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-4038) Create gRPC server processor

[Tony Kurc (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tony Kurc resolved NIFI-4038.
-
Resolution: Fixed

> Create gRPC server processor
> 
>
> Key: NIFI-4038
> URL: https://issues.apache.org/jira/browse/NIFI-4038
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
> Fix For: 1.4.0
>
> Attachments: listen_and_invoke_grpc.xml
>
>
> Create a simple gRPC [1] server processor similar to `HandleHttpRequest` that 
> listens for RPCs from the gRPC processor created in NIFI-4037.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4038) Create gRPC server processor

[Tony Kurc (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tony Kurc updated NIFI-4038:

Fix Version/s: 1.4.0

> Create gRPC server processor
> 
>
> Key: NIFI-4038
> URL: https://issues.apache.org/jira/browse/NIFI-4038
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
> Fix For: 1.4.0
>
> Attachments: listen_and_invoke_grpc.xml
>
>
> Create a simple gRPC [1] server processor similar to `HandleHttpRequest` that 
> listens for RPCs from the gRPC processor created in NIFI-4037.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4038) Create gRPC server processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069354#comment-16069354
 ] 

ASF GitHub Bot commented on NIFI-4038:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1947


> Create gRPC server processor
> 
>
> Key: NIFI-4038
> URL: https://issues.apache.org/jira/browse/NIFI-4038
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
> Attachments: listen_and_invoke_grpc.xml
>
>
> Create a simple gRPC [1] server processor similar to `HandleHttpRequest` that 
> listens for RPCs from the gRPC processor created in NIFI-4037.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4037) Create gRPC client processor

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069352#comment-16069352
 ] 

ASF subversion and git services commented on NIFI-4037:
---

Commit 58a623dfa270a77fa6fdd0fb3ac551eda663d64c in nifi's branch 
refs/heads/master from m-hogue
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=58a623d ]

NIFI-4037 added InvokeGRPC processor, with proto service IDL
NIFI-4038 added ListenGRPC processor

This closes #1947

Signed-off-by: Tony Kurc 


> Create gRPC client processor 
> -
>
> Key: NIFI-4037
> URL: https://issues.apache.org/jira/browse/NIFI-4037
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
> Attachments: nifi_invoke_grpc.png
>
>
> Create a simple gRPC [1] client processor similar to InvokeHTTP that would 
> allow  the sharing of flow files to a configured remote gRPC server.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4038) Create gRPC server processor

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069353#comment-16069353
 ] 

ASF subversion and git services commented on NIFI-4038:
---

Commit 58a623dfa270a77fa6fdd0fb3ac551eda663d64c in nifi's branch 
refs/heads/master from m-hogue
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=58a623d ]

NIFI-4037 added InvokeGRPC processor, with proto service IDL
NIFI-4038 added ListenGRPC processor

This closes #1947

Signed-off-by: Tony Kurc 


> Create gRPC server processor
> 
>
> Key: NIFI-4038
> URL: https://issues.apache.org/jira/browse/NIFI-4038
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
> Attachments: listen_and_invoke_grpc.xml
>
>
> Create a simple gRPC [1] server processor similar to `HandleHttpRequest` that 
> listens for RPCs from the gRPC processor created in NIFI-4037.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1947: NIFI-4038, NIFI-4037 grpc client and server process...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1947


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4105) support the specified Maximum value column and CSV Stream for Cassandra

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069277#comment-16069277
 ] 

ASF GitHub Bot commented on NIFI-4105:
--

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124944247
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
 ---
@@ -213,13 +308,20 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 return;
 }
 }
-
-final ComponentLog logger = getLogger();
 final String selectQuery = 
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(fileToProcess).getValue();
 final long queryTimeout = 
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(fileToProcess).asTimePeriod(TimeUnit.MILLISECONDS);
 final String outputFormat = 
context.getProperty(OUTPUT_FORMAT).getValue();
 final Charset charset = 
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(fileToProcess).getValue());
 final StopWatch stopWatch = new StopWatch(true);
+final String waterMarkDateField = 
context.getProperty(DATE_FIELD).getValue();
+final String tableName = 
context.getProperty(TABLE_NAME).getValue();
+final String keySpace = 
context.getProperty(KEYSPACE).evaluateAttributeExpressions(fileToProcess).getValue();
+
+if ( StringUtils.isEmpty(selectQuery) && 
StringUtils.isEmpty(tableName) ) {
--- End diff --

Yes. you're right.


> support the specified Maximum value column and CSV Stream for Cassandra
> ---
>
> Key: NIFI-4105
> URL: https://issues.apache.org/jira/browse/NIFI-4105
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.3.0
>Reporter: Yoonwon Ko
>
> I'm trying to find a CassandraProcessor to fetch rows whose values in the 
> specified Maximum Value columns are larger than the previously-seen maximum 
> like QueryDatabaseTable.
> But I found only QueryCassandra. It just executes same CQL everytime without 
> keeping maximum value.
> and I think we also need convertToCsvStream option.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1937: NIFI-4105 support the specified Maximum value colum...

[ggthename]

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124944247
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
 ---
@@ -213,13 +308,20 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 return;
 }
 }
-
-final ComponentLog logger = getLogger();
 final String selectQuery = 
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(fileToProcess).getValue();
 final long queryTimeout = 
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(fileToProcess).asTimePeriod(TimeUnit.MILLISECONDS);
 final String outputFormat = 
context.getProperty(OUTPUT_FORMAT).getValue();
 final Charset charset = 
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(fileToProcess).getValue());
 final StopWatch stopWatch = new StopWatch(true);
+final String waterMarkDateField = 
context.getProperty(DATE_FIELD).getValue();
+final String tableName = 
context.getProperty(TABLE_NAME).getValue();
+final String keySpace = 
context.getProperty(KEYSPACE).evaluateAttributeExpressions(fileToProcess).getValue();
+
+if ( StringUtils.isEmpty(selectQuery) && 
StringUtils.isEmpty(tableName) ) {
--- End diff --

Yes. you're right.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (NIFI-3891) Wrap label text in text area and on canvas

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto reassigned NIFI-3891:
---

Assignee: (was: Andy LoPresto)

> Wrap label text in text area and on canvas
> --
>
> Key: NIFI-3891
> URL: https://issues.apache.org/jira/browse/NIFI-3891
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Affects Versions: 1.2.0
>Reporter: Andy LoPresto
>Priority: Minor
>  Labels: beginner, ui
> Attachments: Screen Shot 2017-05-12 at 1.33.28 PM.png, Screen Shot 
> 2017-05-12 at 1.33.33 PM.png
>
>
> The label text does not wrap when editing or displaying the label. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4111) NiFi does not shutdown gracefully

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4111:
-
Assignee: Pierre Villard
  Status: Patch Available  (was: Open)

> NiFi does not shutdown gracefully
> -
>
> Key: NIFI-4111
> URL: https://issues.apache.org/jira/browse/NIFI-4111
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Minor
>
> I don't know exactly for how long we have this issue but NiFi is not able to 
> shutdown gracefully anymore (standalone and cluster setups). It happens even 
> if no processor/CS/RT is running in the instance:
> {noformat}
> 2017-06-22 23:47:40,448 INFO [main] org.apache.nifi.bootstrap.Command Apache 
> NiFi has accepted the Shutdown Command and is shutting down now
> 2017-06-22 23:47:40,527 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:42,540 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:44,553 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:46,569 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:48,585 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:50,601 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:52,614 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:54,626 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:56,640 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:58,655 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:48:00,672 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:48:00,681 WARN [main] org.apache.nifi.bootstrap.Command NiFi 
> has not finished shutting down after 20 seconds. Killing process.
> 2017-06-22 23:48:00,714 INFO [main] org.apache.nifi.bootstrap.Command NiFi 
> has finished shutting down.
> {noformat}
> Thanks to [~markap14], the problem seems to be with shutting down the 
> following thread:
> {noformat}
> 2017-06-21 16:23:35,159 INFO [NiFi logging handler] org.apache.nifi.StdOut 
> "Site-to-Site Worker Thread-1" #87 prio=5 os_prio=31 tid=0x7f9ec968c000 
> nid=0xeb03 waiting on condition [0x000137b4e000]
> 2017-06-21 16:23:35,159 INFO [NiFi logging handler] org.apache.nifi.StdOut
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> 2017-06-21 16:23:35,159 INFO [NiFi logging handler] org.apache.nifi.StdOut
>   at java.lang.Thread.sleep(Native Method)
> 2017-06-21 16:23:35,159 INFO [NiFi logging handler] org.apache.nifi.StdOut
>   at java.lang.Thread.sleep(Thread.java:340)
> 2017-06-21 16:23:35,159 INFO [NiFi logging handler] org.apache.nifi.StdOut
>   at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
> 2017-06-21 16:23:35,159 INFO [NiFi logging handler] org.apache.nifi.StdOut
>   at 
> org.apache.nifi.remote.io.socket.SocketChannelInputStream.read(SocketChannelInputStream.java:120)
> 2017-06-21 16:23:35,159 INFO [NiFi logging handler] org.apache.nifi.StdOut
>   at 
> org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:51)
> 2017-06-21 16:23:35,160 INFO [NiFi logging handler] org.apache.nifi.StdOut
>   at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> 2017-06-21 16:23:35,160 INFO [NiFi logging handler] org.apache.nifi.StdOut
>   at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> 2017-06-21 16:23:35,160 INFO [NiFi logging handler] org.apache.nifi.StdOut
>   - locked <0x0007be373b78> (a 
> org.apache.nifi.stream.io.BufferedInputStream)
> 2017-06-21 16:23:35,160 INFO [NiFi logging handler] org.apache.nifi.StdOut
>   at 
> org.apache.nifi.remote.io.InterruptableInputStream.read(InterruptableInputStream.java:39)
> 2017-06-21 16:23:35,160 INFO [NiFi logging handler] org.apache.nifi.StdOut
>   at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:337)
> 2017-06-21 16:23:35,160 INFO [NiFi logging handler] org.apache.nifi.StdOut
>   at java.io.DataInputStream.readUTF(DataInputStream.java:589)
> 2017-06-21 16:23:35,160 INFO [NiFi logging handler] org.apache.nifi.StdOut
>   at 

[jira] [Commented] (NIFI-4111) NiFi does not shutdown gracefully

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069066#comment-16069066
 ] 

ASF GitHub Bot commented on NIFI-4111:
--

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1963

NIFI-4111 - NiFi shutdown

Fixed threads shutdown so that NiFi can shutdown gracefully

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4111

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1963.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1963


commit 67ba3f5d9889120f223812fb4eb533d4a4214f46
Author: Pierre Villard 
Date:   2017-06-29T22:03:53Z

NIFI-4111 - NiFi shutdown

Fixed threads shutdown so that NiFi can shutdown gracefully




> NiFi does not shutdown gracefully
> -
>
> Key: NIFI-4111
> URL: https://issues.apache.org/jira/browse/NIFI-4111
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Pierre Villard
>Priority: Minor
>
> I don't know exactly for how long we have this issue but NiFi is not able to 
> shutdown gracefully anymore (standalone and cluster setups). It happens even 
> if no processor/CS/RT is running in the instance:
> {noformat}
> 2017-06-22 23:47:40,448 INFO [main] org.apache.nifi.bootstrap.Command Apache 
> NiFi has accepted the Shutdown Command and is shutting down now
> 2017-06-22 23:47:40,527 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:42,540 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:44,553 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:46,569 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:48,585 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:50,601 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:52,614 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:54,626 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:56,640 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:47:58,655 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 2017-06-22 23:48:00,672 INFO [main] org.apache.nifi.bootstrap.Command Waiting 
> for Apache NiFi to finish shutting down...
> 

[GitHub] nifi pull request #1963: NIFI-4111 - NiFi shutdown

[pvillard31]

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1963

NIFI-4111 - NiFi shutdown

Fixed threads shutdown so that NiFi can shutdown gracefully

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4111

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1963.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1963


commit 67ba3f5d9889120f223812fb4eb533d4a4214f46
Author: Pierre Villard 
Date:   2017-06-29T22:03:53Z

NIFI-4111 - NiFi shutdown

Fixed threads shutdown so that NiFi can shutdown gracefully




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4060) Create a MergeRecord Processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068918#comment-16068918
 ] 

ASF GitHub Bot commented on NIFI-4060:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1958#discussion_r124901819
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/MergeRecord.java
 ---
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.FragmentAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.FlowFileFilters;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.merge.AttributeStrategyUtil;
+import org.apache.nifi.processors.standard.merge.RecordBinManager;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+
+@SideEffectFree
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"merge", "record", "content", "correlation", "stream", "event"})
+@CapabilityDescription("This Processor merges together multiple 
record-oriented FlowFiles into a single FlowFile that contains all of the 
Records of the input FlowFiles. "
++ "This Processor works by creating 'bins' and then adding FlowFiles 
to these bins until they are full. Once a bin is full, all of the FlowFiles 
will be combined into "
++ "a single output FlowFile, and that FlowFile will be routed to the 
'merged' Relationship. A bin will consist of potentially many 'like FlowFiles'. 
In order for two "
++ "FlowFiles to be considered 'like FlowFiles', they must have the 
same Schema (as identified by the Record Reader) and, if the  property "
++ "is set, the same value for the specified attribute. See Processor 
Usage and Additional Details for more information.")
+@ReadsAttributes({
+

[GitHub] nifi pull request #1958: NIFI-4060: Initial implementation of MergeRecord

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1958#discussion_r124901819
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/MergeRecord.java
 ---
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.FragmentAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.FlowFileFilters;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.merge.AttributeStrategyUtil;
+import org.apache.nifi.processors.standard.merge.RecordBinManager;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+
+@SideEffectFree
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"merge", "record", "content", "correlation", "stream", "event"})
+@CapabilityDescription("This Processor merges together multiple 
record-oriented FlowFiles into a single FlowFile that contains all of the 
Records of the input FlowFiles. "
++ "This Processor works by creating 'bins' and then adding FlowFiles 
to these bins until they are full. Once a bin is full, all of the FlowFiles 
will be combined into "
++ "a single output FlowFile, and that FlowFile will be routed to the 
'merged' Relationship. A bin will consist of potentially many 'like FlowFiles'. 
In order for two "
++ "FlowFiles to be considered 'like FlowFiles', they must have the 
same Schema (as identified by the Record Reader) and, if the  property "
++ "is set, the same value for the specified attribute. See Processor 
Usage and Additional Details for more information.")
+@ReadsAttributes({
+@ReadsAttribute(attribute = "fragment.identifier", description = 
"Applicable only if the  property is set to Defragment. "
++ "All FlowFiles with the same value for this attribute will be 
bundled together."),
+@ReadsAttribute(attribute = 

[jira] [Commented] (NIFIREG-4) Add new logo to Registry web page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068838#comment-16068838
 ] 

ASF GitHub Bot commented on NIFIREG-4:
--

GitHub user andrewmlim opened a pull request:

https://github.com/apache/nifi-site/pull/20

NIFIREG-4 Add/Update logos on Registry and MiNiFi web pages

- New Registry logo added
- Updated MiNiFi logos (now have Apache NiFi added)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewmlim/nifi-site NIFIREG-4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-site/pull/20.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20


commit a8b5f91a77546c334293f5a890eb10a9f1e91ec9
Author: Andrew Lim 
Date:   2017-06-29T19:33:50Z

NIFIREG-4 Add/Update logos on Registry and MiNiFi web pages




> Add new logo to Registry web page
> -
>
> Key: NIFIREG-4
> URL: https://issues.apache.org/jira/browse/NIFIREG-4
> Project: NiFi Registry
>  Issue Type: Improvement
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> A new Registry logo is available 
> (https://issues.apache.org/jira/browse/NIFIREG-2)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-site pull request #20: NIFIREG-4 Add/Update logos on Registry and MiNiF...

[andrewmlim]

GitHub user andrewmlim opened a pull request:

https://github.com/apache/nifi-site/pull/20

NIFIREG-4 Add/Update logos on Registry and MiNiFi web pages

- New Registry logo added
- Updated MiNiFi logos (now have Apache NiFi added)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewmlim/nifi-site NIFIREG-4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-site/pull/20.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20


commit a8b5f91a77546c334293f5a890eb10a9f1e91ec9
Author: Andrew Lim 
Date:   2017-06-29T19:33:50Z

NIFIREG-4 Add/Update logos on Registry and MiNiFi web pages




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4060) Create a MergeRecord Processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068803#comment-16068803
 ] 

ASF GitHub Bot commented on NIFI-4060:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/1958
  
Reviewing...


> Create a MergeRecord Processor
> --
>
> Key: NIFI-4060
> URL: https://issues.apache.org/jira/browse/NIFI-4060
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> When record-oriented data is received one record or a time or needs to be 
> split into small chunks for one reason or another, it will be helpful to be 
> able to combine those records into a single FlowFile that is made up of many 
> records for efficiency purposes, or to deliver to downstream systems as 
> larger batches. This processor should function similarly to MergeContent but 
> make use of Record Readers and Record Writer so that users don't have to deal 
> with headers, footers, demarcators, etc.
> The Processor will also need to ensure that records only get merge into the 
> same FlowFile if they have compatible schemas.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1958: NIFI-4060: Initial implementation of MergeRecord

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/1958
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (NIFI-3880) Fully document TLS Toolkit options in Admin Guide

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim reassigned NIFI-3880:


Assignee: Andrew Lim

> Fully document TLS Toolkit options in Admin Guide
> -
>
> Key: NIFI-3880
> URL: https://issues.apache.org/jira/browse/NIFI-3880
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.2.0
>Reporter: Andy LoPresto
>Assignee: Andrew Lim
>Priority: Minor
>  Labels: documentation, tls-toolkit
>
> The TLS Toolkit has "common options" described in the [Admin 
> Guide|https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls-generation-toolkit]
>  but the complete list of options is not available without running the tool 
> locally. The comprehensive list of options and their descriptions should be 
> made available for reference in the guide. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-4125) Add basic security settings to TransformXml

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto resolved NIFI-4125.
-
Resolution: Fixed

> Add basic security settings to TransformXml
> ---
>
> Key: NIFI-4125
> URL: https://issues.apache.org/jira/browse/NIFI-4125
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Yuri
>Priority: Minor
>  Labels: newbie, security, xslt
>
> Since data flows can generally deal with non-trusted data, the processors 
> should handle it in a secure manner.
> In case of XML there are various known vulnerabilities - 
> [OWASP|https://www.owasp.org/index.php/XML_External_Entity_%28XXE%29_Processing].
>  Some can be mitigated via XML parser/XSLT Processor features.
> The TransformXml processor should have a setting enabling these secure 
> settings.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4143) Make configurable maximum number of concurrent requests

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4143:
-
Status: Patch Available  (was: Open)

> Make configurable maximum number of concurrent requests
> ---
>
> Key: NIFI-4143
> URL: https://issues.apache.org/jira/browse/NIFI-4143
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> At the moment, the maximum number of concurrent requests is hard coded in 
> {{ThreadPoolRequestReplicator}}
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ThreadPoolRequestReplicator.java
> The value is equal to 100.
> In some situations where multiple factors are combined (large cluster, S2S to 
> load balance data in the cluster, multiple users accessing the UI), the limit 
> can be reached and the UI may become intermittently unavailable with the 
> message: "There are too many outstanding HTTP requests with a total 100 
> outstanding requests".
> This value should be configurable in nifi.properties allowing users to 
> increase the value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4143) Make configurable maximum number of concurrent requests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068753#comment-16068753
 ] 

ASF GitHub Bot commented on NIFI-4143:
--

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1962

NIFI-4143 - externalize MAX_CONCURRENT_REQUESTS

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4143

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1962.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1962


commit 11735b05ace6f862fe2339fe66cca4909a31edbd
Author: Pierre Villard 
Date:   2017-06-29T18:27:55Z

NIFI-4143 - externalize MAX_CONCURRENT_REQUESTS




> Make configurable maximum number of concurrent requests
> ---
>
> Key: NIFI-4143
> URL: https://issues.apache.org/jira/browse/NIFI-4143
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> At the moment, the maximum number of concurrent requests is hard coded in 
> {{ThreadPoolRequestReplicator}}
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ThreadPoolRequestReplicator.java
> The value is equal to 100.
> In some situations where multiple factors are combined (large cluster, S2S to 
> load balance data in the cluster, multiple users accessing the UI), the limit 
> can be reached and the UI may become intermittently unavailable with the 
> message: "There are too many outstanding HTTP requests with a total 100 
> outstanding requests".
> This value should be configurable in nifi.properties allowing users to 
> increase the value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1962: NIFI-4143 - externalize MAX_CONCURRENT_REQUESTS

[pvillard31]

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/1962

NIFI-4143 - externalize MAX_CONCURRENT_REQUESTS

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4143

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1962.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1962


commit 11735b05ace6f862fe2339fe66cca4909a31edbd
Author: Pierre Villard 
Date:   2017-06-29T18:27:55Z

NIFI-4143 - externalize MAX_CONCURRENT_REQUESTS




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4125) Add basic security settings to TransformXml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068750#comment-16068750
 ] 

ASF GitHub Bot commented on NIFI-4125:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/1946
  
Thanks for that detailed explanation. I updated the commit message to fit 
our format. 

Ran `contrib-check` and all tests pass. +1, merging. 


> Add basic security settings to TransformXml
> ---
>
> Key: NIFI-4125
> URL: https://issues.apache.org/jira/browse/NIFI-4125
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Yuri
>Priority: Minor
>  Labels: newbie, security, xslt
>
> Since data flows can generally deal with non-trusted data, the processors 
> should handle it in a secure manner.
> In case of XML there are various known vulnerabilities - 
> [OWASP|https://www.owasp.org/index.php/XML_External_Entity_%28XXE%29_Processing].
>  Some can be mitigated via XML parser/XSLT Processor features.
> The TransformXml processor should have a setting enabling these secure 
> settings.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4125) Add basic security settings to TransformXml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068751#comment-16068751
 ] 

ASF GitHub Bot commented on NIFI-4125:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1946


> Add basic security settings to TransformXml
> ---
>
> Key: NIFI-4125
> URL: https://issues.apache.org/jira/browse/NIFI-4125
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Yuri
>Priority: Minor
>  Labels: newbie, security, xslt
>
> Since data flows can generally deal with non-trusted data, the processors 
> should handle it in a secure manner.
> In case of XML there are various known vulnerabilities - 
> [OWASP|https://www.owasp.org/index.php/XML_External_Entity_%28XXE%29_Processing].
>  Some can be mitigated via XML parser/XSLT Processor features.
> The TransformXml processor should have a setting enabling these secure 
> settings.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFIREG-4) Addto Registry web page

[Andrew Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068748#comment-16068748
 ] 

Andrew Lim commented on NIFIREG-4:
--

Also, the MiNiFi logo was recently updated 
(https://issues.apache.org/jira/browse/MINIFI-32).  Will fix that as well.

> Addto Registry web page
> ---
>
> Key: NIFIREG-4
> URL: https://issues.apache.org/jira/browse/NIFIREG-4
> Project: NiFi Registry
>  Issue Type: Improvement
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> A new Registry logo is available 
> (https://issues.apache.org/jira/browse/NIFIREG-2)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4125) Add basic security settings to TransformXml

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068749#comment-16068749
 ] 

ASF subversion and git services commented on NIFI-4125:
---

Commit 3bf1d127062a2d52d7be32e5ef29e19242219f48 in nifi's branch 
refs/heads/master from [~yuri1969]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=3bf1d12 ]

NIFI-4125 Added secure transform feature and configuration to TransformXML 
processor to mitigate XXE file system leaks.

This closes #1946.

Signed-off-by: Andy LoPresto 


> Add basic security settings to TransformXml
> ---
>
> Key: NIFI-4125
> URL: https://issues.apache.org/jira/browse/NIFI-4125
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Yuri
>Priority: Minor
>  Labels: newbie, security, xslt
>
> Since data flows can generally deal with non-trusted data, the processors 
> should handle it in a secure manner.
> In case of XML there are various known vulnerabilities - 
> [OWASP|https://www.owasp.org/index.php/XML_External_Entity_%28XXE%29_Processing].
>  Some can be mitigated via XML parser/XSLT Processor features.
> The TransformXml processor should have a setting enabling these secure 
> settings.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1946: NIFI-4125 - Add basic security settings to Transfor...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1946


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1946: NIFI-4125 - Add basic security settings to TransformXml

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/1946
  
Thanks for that detailed explanation. I updated the commit message to fit 
our format. 

Ran `contrib-check` and all tests pass. +1, merging. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFIREG-4) Addto Registry web page

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFIREG-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim updated NIFIREG-4:
-
Summary: Addto Registry web page  (was: Add/update logos on MiNiFi and 
Registry web pages)

> Addto Registry web page
> ---
>
> Key: NIFIREG-4
> URL: https://issues.apache.org/jira/browse/NIFIREG-4
> Project: NiFi Registry
>  Issue Type: Improvement
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> A new Registry logo is available 
> (https://issues.apache.org/jira/browse/NIFIREG-2)
> Also, the MiNiFi logo was recently updated 
> (https://issues.apache.org/jira/browse/MINIFI-32)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFIREG-4) Addto Registry web page

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFIREG-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim updated NIFIREG-4:
-
Description: A new Registry logo is available 
(https://issues.apache.org/jira/browse/NIFIREG-2)  (was: A new Registry logo is 
available (https://issues.apache.org/jira/browse/NIFIREG-2)

Also, the MiNiFi logo was recently updated 
(https://issues.apache.org/jira/browse/MINIFI-32))

> Addto Registry web page
> ---
>
> Key: NIFIREG-4
> URL: https://issues.apache.org/jira/browse/NIFIREG-4
> Project: NiFi Registry
>  Issue Type: Improvement
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> A new Registry logo is available 
> (https://issues.apache.org/jira/browse/NIFIREG-2)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFIREG-4) Add/update logos on MiNiFi and Registry web pages

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFIREG-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim updated NIFIREG-4:
-
Description: 
A new Registry logo is available 
(https://issues.apache.org/jira/browse/NIFIREG-2)

Also, the MiNiFi logo was recently updated 
(https://issues.apache.org/jira/browse/MINIFI-32)

> Add/update logos on MiNiFi and Registry web pages
> -
>
> Key: NIFIREG-4
> URL: https://issues.apache.org/jira/browse/NIFIREG-4
> Project: NiFi Registry
>  Issue Type: Improvement
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> A new Registry logo is available 
> (https://issues.apache.org/jira/browse/NIFIREG-2)
> Also, the MiNiFi logo was recently updated 
> (https://issues.apache.org/jira/browse/MINIFI-32)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFIREG-4) Add/update logos on MiNiFi and Registry web pages

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFIREG-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim updated NIFIREG-4:
-
Summary: Add/update logos on MiNiFi and Registry web pages  (was: Add logo 
to Registry web page)

> Add/update logos on MiNiFi and Registry web pages
> -
>
> Key: NIFIREG-4
> URL: https://issues.apache.org/jira/browse/NIFIREG-4
> Project: NiFi Registry
>  Issue Type: Improvement
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-4143) Make configurable maximum number of concurrent requests

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard reassigned NIFI-4143:


Assignee: Pierre Villard

> Make configurable maximum number of concurrent requests
> ---
>
> Key: NIFI-4143
> URL: https://issues.apache.org/jira/browse/NIFI-4143
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> At the moment, the maximum number of concurrent requests is hard coded in 
> {{ThreadPoolRequestReplicator}}
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ThreadPoolRequestReplicator.java
> The value is equal to 100.
> In some situations where multiple factors are combined (large cluster, S2S to 
> load balance data in the cluster, multiple users accessing the UI), the limit 
> can be reached and the UI may become intermittently unavailable with the 
> message: "There are too many outstanding HTTP requests with a total 100 
> outstanding requests".
> This value should be configurable in nifi.properties allowing users to 
> increase the value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFIREG-4) Add logo to Registry web page

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFIREG-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim updated NIFIREG-4:
-
Summary: Add logo to Registry web page  (was: Add logo to header of 
Registry web page)

> Add logo to Registry web page
> -
>
> Key: NIFIREG-4
> URL: https://issues.apache.org/jira/browse/NIFIREG-4
> Project: NiFi Registry
>  Issue Type: Improvement
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4136) GrokReader - Add a failure option to unmatch behavior options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068651#comment-16068651
 ] 

ASF GitHub Bot commented on NIFI-4136:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1955#discussion_r124861720
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/grok/GrokReader.java
 ---
@@ -74,6 +74,8 @@
 "The line of text that does not match the Grok Expression will be 
appended to the last field of the prior message.");
 static final AllowableValue SKIP_LINE = new 
AllowableValue("skip-line", "Skip Line",
 "The line of text that does not match the Grok Expression will be 
skipped.");
+static final AllowableValue ROUTE_TO_FAILURE = new 
AllowableValue("route-to-failure", "Route to failure",
--- End diff --

Yep, that's fair.


> GrokReader - Add a failure option to unmatch behavior options
> -
>
> Key: NIFI-4136
> URL: https://issues.apache.org/jira/browse/NIFI-4136
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> At the moment, when using the GrokReader, if a line does not match the grok 
> expression (and is not part of a stack trace), the line can be either ignored 
> (the line will be completely skipped) or  appended to the last field from the 
> previous line.
> In the case where appending is not desired and that data should not be 
> ignored/deleted, we should add the option to route the full flow file to the 
> failure relationship. This way the flow file could be treated in a different 
> way (for example with SplitText and ExtractGrok to isolate the incorrect 
> lines and re-route the correct lines back to the Record processors).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1955: NIFI-4136 Add a failure option to unmatch behavior ...

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1955#discussion_r124861720
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/grok/GrokReader.java
 ---
@@ -74,6 +74,8 @@
 "The line of text that does not match the Grok Expression will be 
appended to the last field of the prior message.");
 static final AllowableValue SKIP_LINE = new 
AllowableValue("skip-line", "Skip Line",
 "The line of text that does not match the Grok Expression will be 
skipped.");
+static final AllowableValue ROUTE_TO_FAILURE = new 
AllowableValue("route-to-failure", "Route to failure",
--- End diff --

Yep, that's fair.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4024) Create EvaluateRecordPath processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068638#comment-16068638
 ] 

ASF GitHub Bot commented on NIFI-4024:
--

GitHub user MikeThomsen opened a pull request:

https://github.com/apache/nifi/pull/1961

NIFI-4024 Added org.apache.nifi.hbase.PutHBaseRecord

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeThomsen/nifi NIFI-4024

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1961.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1961


commit f624e69f88fd84e619a5044abfda250a755d96d6
Author: Mike Thomsen 
Date:   2017-06-23T11:50:26Z

NIFI-4024 Added org.apache.nifi.hbase.PutHBaseRecord




> Create EvaluateRecordPath processor
> ---
>
> Key: NIFI-4024
> URL: https://issues.apache.org/jira/browse/NIFI-4024
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Steve Champagne
>Priority: Minor
>
> With the new RecordPath DSL, it would be nice if there was a processor that 
> could pull fields into attributes of the flowfile based on a RecordPath. This 
> would be similar to the EvaluateJsonPath processor that currently exists, 
> except it could be used to pull fields from arbitrary record formats. My 
> current use case for it would be pulling fields out of Avro records while 
> skipping the steps of having to convert Avro to JSON, evaluate JsonPath, and 
> then converting back to Avro. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1961: NIFI-4024 Added org.apache.nifi.hbase.PutHBaseRecor...

[MikeThomsen]

GitHub user MikeThomsen opened a pull request:

https://github.com/apache/nifi/pull/1961

NIFI-4024 Added org.apache.nifi.hbase.PutHBaseRecord

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeThomsen/nifi NIFI-4024

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1961.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1961


commit f624e69f88fd84e619a5044abfda250a755d96d6
Author: Mike Thomsen 
Date:   2017-06-23T11:50:26Z

NIFI-4024 Added org.apache.nifi.hbase.PutHBaseRecord




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-4143) Make configurable maximum number of concurrent requests

[Pierre Villard (JIRA)]

Pierre Villard created NIFI-4143:


 Summary: Make configurable maximum number of concurrent requests
 Key: NIFI-4143
 URL: https://issues.apache.org/jira/browse/NIFI-4143
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework
Reporter: Pierre Villard


At the moment, the maximum number of concurrent requests is hard coded in 
{{ThreadPoolRequestReplicator}}

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ThreadPoolRequestReplicator.java

The value is equal to 100.

In some situations where multiple factors are combined (large cluster, S2S to 
load balance data in the cluster, multiple users accessing the UI), the limit 
can be reached and the UI may become intermittently unavailable with the 
message: "There are too many outstanding HTTP requests with a total 100 
outstanding requests".

This value should be configurable in nifi.properties allowing users to increase 
the value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFIREG-4) Add logo to header of Registry web page

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFIREG-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim reassigned NIFIREG-4:


Assignee: Andrew Lim

> Add logo to header of Registry web page
> ---
>
> Key: NIFIREG-4
> URL: https://issues.apache.org/jira/browse/NIFIREG-4
> Project: NiFi Registry
>  Issue Type: Improvement
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFIREG-4) Add logo to header of Registry web page

[Andrew Lim (JIRA)]

Andrew Lim created NIFIREG-4:


 Summary: Add logo to header of Registry web page
 Key: NIFIREG-4
 URL: https://issues.apache.org/jira/browse/NIFIREG-4
 Project: NiFi Registry
  Issue Type: Improvement
Reporter: Andrew Lim
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4138) Add Component Alignment to User Guide

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim updated NIFI-4138:
-
Fix Version/s: 1.4.0

> Add Component Alignment to User Guide
> -
>
> Key: NIFI-4138
> URL: https://issues.apache.org/jira/browse/NIFI-4138
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.2.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
> Fix For: 1.4.0
>
>
> Component alignment was added in 1.2.0 (NIFI-96).  Should add this new 
> feature to the User Guide.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4138) Add Component Alignment to User Guide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068526#comment-16068526
 ] 

ASF GitHub Bot commented on NIFI-4138:
--

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/1959
  
Thanks @andrewmlim this has been merged to master.


> Add Component Alignment to User Guide
> -
>
> Key: NIFI-4138
> URL: https://issues.apache.org/jira/browse/NIFI-4138
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.2.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> Component alignment was added in 1.2.0 (NIFI-96).  Should add this new 
> feature to the User Guide.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4138) Add Component Alignment to User Guide

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068522#comment-16068522
 ] 

ASF subversion and git services commented on NIFI-4138:
---

Commit 3089d9dce00e04d94572245f9ac65fd936615c88 in nifi's branch 
refs/heads/master from [~andrewmlim]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=3089d9d ]

NIFI-4138 Add Component Alignment to User Guide

Signed-off-by: Scott Aslan 

This closes #1959


> Add Component Alignment to User Guide
> -
>
> Key: NIFI-4138
> URL: https://issues.apache.org/jira/browse/NIFI-4138
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.2.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> Component alignment was added in 1.2.0 (NIFI-96).  Should add this new 
> feature to the User Guide.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1959: NIFI-4138 Add Component Alignment to User Guide

[scottyaslan]

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/1959
  
Thanks @andrewmlim this has been merged to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (NIFI-4138) Add Component Alignment to User Guide

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan resolved NIFI-4138.
---
Resolution: Fixed

> Add Component Alignment to User Guide
> -
>
> Key: NIFI-4138
> URL: https://issues.apache.org/jira/browse/NIFI-4138
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.2.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> Component alignment was added in 1.2.0 (NIFI-96).  Should add this new 
> feature to the User Guide.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4138) Add Component Alignment to User Guide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068523#comment-16068523
 ] 

ASF GitHub Bot commented on NIFI-4138:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1959


> Add Component Alignment to User Guide
> -
>
> Key: NIFI-4138
> URL: https://issues.apache.org/jira/browse/NIFI-4138
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.2.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> Component alignment was added in 1.2.0 (NIFI-96).  Should add this new 
> feature to the User Guide.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4105) support the specified Maximum value column and CSV Stream for Cassandra

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068521#comment-16068521
 ] 

ASF GitHub Bot commented on NIFI-4105:
--

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124841474
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
 ---
@@ -75,19 +81,64 @@
 @Tags({"cassandra", "cql", "select"})
 @EventDriven
 @InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
-@CapabilityDescription("Execute provided Cassandra Query Language (CQL) 
select query on a Cassandra 1.x, 2.x, or 3.0.x cluster. Query result "
-+ "may be converted to Avro or JSON format. Streaming is used so 
arbitrarily large result sets are supported. This processor can be "
+@CapabilityDescription("Executes provided Cassandra Query Language (CQL) 
select query on a Cassandra to fetch all rows whose values"
--- End diff --

ORDER BY and ROWNUM


> support the specified Maximum value column and CSV Stream for Cassandra
> ---
>
> Key: NIFI-4105
> URL: https://issues.apache.org/jira/browse/NIFI-4105
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.3.0
>Reporter: Yoonwon Ko
>
> I'm trying to find a CassandraProcessor to fetch rows whose values in the 
> specified Maximum Value columns are larger than the previously-seen maximum 
> like QueryDatabaseTable.
> But I found only QueryCassandra. It just executes same CQL everytime without 
> keeping maximum value.
> and I think we also need convertToCsvStream option.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1959: NIFI-4138 Add Component Alignment to User Guide

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1959


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1937: NIFI-4105 support the specified Maximum value colum...

[ggthename]

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124841474
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
 ---
@@ -75,19 +81,64 @@
 @Tags({"cassandra", "cql", "select"})
 @EventDriven
 @InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
-@CapabilityDescription("Execute provided Cassandra Query Language (CQL) 
select query on a Cassandra 1.x, 2.x, or 3.0.x cluster. Query result "
-+ "may be converted to Avro or JSON format. Streaming is used so 
arbitrarily large result sets are supported. This processor can be "
+@CapabilityDescription("Executes provided Cassandra Query Language (CQL) 
select query on a Cassandra to fetch all rows whose values"
--- End diff --

ORDER BY and ROWNUM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3897) Queue with flow files thinks it is empty

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068482#comment-16068482
 ] 

ASF GitHub Bot commented on NIFI-3897:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/1960

NIFI-3897: If swapping data into queue and an unexpected exception/er…

…ror is thrown, do not lose track of the swap file

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-3897

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1960.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1960


commit 97ad4fa824a84285512cd1549e73fd98711190f4
Author: Mark Payne 
Date:   2017-06-29T14:49:34Z

NIFI-3897: If swapping data into queue and an unexpected exception/error is 
thrown, do not lose track of the swap file




> Queue with flow files thinks it is empty
> 
>
> Key: NIFI-3897
> URL: https://issues.apache.org/jira/browse/NIFI-3897
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Joseph Gresock
>Assignee: Mark Payne
> Fix For: 1.4.0
>
> Attachments: queue-has-no-flowfiles.png
>
>
> A particular queue is listed as having 80,000 flow files, but NiFi thinks 
> there are none to list.  I was originally inclined to believe that the queue 
> is indeed empty, and that there is somehow a pointer to 80,000 flow files 
> leftover on the queue.
> Unfortunately, the logs have rolled over so I can't see what error or warning 
> messages may have happened at the time, and I can't reproduce it reliably.
> Update: I happened to reboot my cluster, and suddenly the 80,000 flow files 
> went through the flow.  This makes the problem considerably worse, since it 
> appears the flow files were "invisible" until the reboot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-3897) Queue with flow files thinks it is empty

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-3897:
-
Fix Version/s: 1.4.0
   Status: Patch Available  (was: Open)

> Queue with flow files thinks it is empty
> 
>
> Key: NIFI-3897
> URL: https://issues.apache.org/jira/browse/NIFI-3897
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Joseph Gresock
>Assignee: Mark Payne
> Fix For: 1.4.0
>
> Attachments: queue-has-no-flowfiles.png
>
>
> A particular queue is listed as having 80,000 flow files, but NiFi thinks 
> there are none to list.  I was originally inclined to believe that the queue 
> is indeed empty, and that there is somehow a pointer to 80,000 flow files 
> leftover on the queue.
> Unfortunately, the logs have rolled over so I can't see what error or warning 
> messages may have happened at the time, and I can't reproduce it reliably.
> Update: I happened to reboot my cluster, and suddenly the 80,000 flow files 
> went through the flow.  This makes the problem considerably worse, since it 
> appears the flow files were "invisible" until the reboot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1960: NIFI-3897: If swapping data into queue and an unexp...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/1960

NIFI-3897: If swapping data into queue and an unexpected exception/er…

…ror is thrown, do not lose track of the swap file

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-3897

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1960.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1960


commit 97ad4fa824a84285512cd1549e73fd98711190f4
Author: Mark Payne 
Date:   2017-06-29T14:49:34Z

NIFI-3897: If swapping data into queue and an unexpected exception/error is 
thrown, do not lose track of the swap file




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3503) Create a 'SplitCSV' processor

[Wesley L Lawrence (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068480#comment-16068480
 ] 

Wesley L Lawrence commented on NIFI-3503:
-

I've been using the SplitRecord processor, and I also think it's sufficient.

> Create a 'SplitCSV' processor
> -
>
> Key: NIFI-3503
> URL: https://issues.apache.org/jira/browse/NIFI-3503
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Wesley L Lawrence
>Priority: Minor
>
> While the 'SplitText' processor helps break up newline separated records into 
> individual files, it's not uncommon to have CSV files where records span 
> multiple lines, and 'SplitText' isn't able or meant to handle this.
> Currently, one can replace, remove, or escape newline characters that exist 
> in a single CSV record by searching within quoted columns with 'ReplaceText', 
> before passing the data onto 'SplitText'. However, this may not work in all 
> cases, or could potentially remove the valid newline character at the end of 
> a CSV record, if all edge cases aren't properly covered with regex.
> Having a dedicated 'SplitCSV' processor will solve this problem, and be a 
> simpler approach for users.
> See the following [Apache NiFi user email 
> thread|https://mail-archives.apache.org/mod_mbox/nifi-users/201702.mbox/%3CCAFuL2BbgymFXwu5fRyd8pP-zu6WkToqPE2Ek7bkyBg0_-cknqQ%40mail.gmail.com%3E]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-minifi-cpp pull request #116: Minifi 341 - Tailfile Delimiter for input

[jdye64]

GitHub user jdye64 opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/116

Minifi 341 - Tailfile Delimiter for input

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [ ] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jdye64/nifi-minifi-cpp MINIFI-341

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/116.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #116


commit 15f571980ccce197202d9292de429f203ff6b67a
Author: Jeremy Dyer 
Date:   2017-06-23T02:29:21Z

MINIFI-341. Introduce delimiter to TailFile to delimit incoming data 
appropriately.

commit 37741406d82e969ab8eeaabd5df196ad3de6eb5a
Author: Jeremy Dyer 
Date:   2017-06-23T13:09:00Z

Input delimiter changed from std::string to char

commit b8bdd639ad261a16a083a7ce830de7f425e83bcc
Author: Jeremy Dyer 
Date:   2017-06-28T19:24:33Z

Updates to TailFile delimiter logic

commit 2491a70fa2538859995c5ef82578c7a3ec112f94
Author: Jeremy Dyer 
Date:   2017-06-29T15:24:01Z

Updates for TailFile delimiter




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4138) Add Component Alignment to User Guide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068468#comment-16068468
 ] 

ASF GitHub Bot commented on NIFI-4138:
--

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/1959
  
Reviewing


> Add Component Alignment to User Guide
> -
>
> Key: NIFI-4138
> URL: https://issues.apache.org/jira/browse/NIFI-4138
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.2.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> Component alignment was added in 1.2.0 (NIFI-96).  Should add this new 
> feature to the User Guide.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1959: NIFI-4138 Add Component Alignment to User Guide

[scottyaslan]

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/1959
  
Reviewing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4138) Add Component Alignment to User Guide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068464#comment-16068464
 ] 

ASF GitHub Bot commented on NIFI-4138:
--

GitHub user andrewmlim opened a pull request:

https://github.com/apache/nifi/pull/1959

NIFI-4138 Add Component Alignment to User Guide




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewmlim/nifi NIFI-4138

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1959.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1959


commit 6e5fe62d8da04acaf47d4846c18416836de37ee4
Author: Andrew Lim 
Date:   2017-06-29T15:17:54Z

NIFI-4138 Add Component Alignment to User Guide




> Add Component Alignment to User Guide
> -
>
> Key: NIFI-4138
> URL: https://issues.apache.org/jira/browse/NIFI-4138
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.2.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> Component alignment was added in 1.2.0 (NIFI-96).  Should add this new 
> feature to the User Guide.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1959: NIFI-4138 Add Component Alignment to User Guide

[andrewmlim]

GitHub user andrewmlim opened a pull request:

https://github.com/apache/nifi/pull/1959

NIFI-4138 Add Component Alignment to User Guide




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewmlim/nifi NIFI-4138

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1959.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1959


commit 6e5fe62d8da04acaf47d4846c18416836de37ee4
Author: Andrew Lim 
Date:   2017-06-29T15:17:54Z

NIFI-4138 Add Component Alignment to User Guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (NIFI-3897) Queue with flow files thinks it is empty

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne reassigned NIFI-3897:


Assignee: Mark Payne

> Queue with flow files thinks it is empty
> 
>
> Key: NIFI-3897
> URL: https://issues.apache.org/jira/browse/NIFI-3897
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Joseph Gresock
>Assignee: Mark Payne
> Attachments: queue-has-no-flowfiles.png
>
>
> A particular queue is listed as having 80,000 flow files, but NiFi thinks 
> there are none to list.  I was originally inclined to believe that the queue 
> is indeed empty, and that there is somehow a pointer to 80,000 flow files 
> leftover on the queue.
> Unfortunately, the logs have rolled over so I can't see what error or warning 
> messages may have happened at the time, and I can't reproduce it reliably.
> Update: I happened to reboot my cluster, and suddenly the 80,000 flow files 
> went through the flow.  This makes the problem considerably worse, since it 
> appears the flow files were "invisible" until the reboot.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1955: NIFI-4136 Add a failure option to unmatch behavior ...

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1955#discussion_r124821493
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/grok/GrokReader.java
 ---
@@ -74,6 +74,8 @@
 "The line of text that does not match the Grok Expression will be 
appended to the last field of the prior message.");
 static final AllowableValue SKIP_LINE = new 
AllowableValue("skip-line", "Skip Line",
 "The line of text that does not match the Grok Expression will be 
skipped.");
+static final AllowableValue ROUTE_TO_FAILURE = new 
AllowableValue("route-to-failure", "Route to failure",
--- End diff --

Since it is up to the processor using the GrokReader to determine if 
something will be routed to failure, should we call this "Failure" or "Produce 
Error" or something indicating that reader is going to throw an exception? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4136) GrokReader - Add a failure option to unmatch behavior options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068431#comment-16068431
 ] 

ASF GitHub Bot commented on NIFI-4136:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1955#discussion_r124821493
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/grok/GrokReader.java
 ---
@@ -74,6 +74,8 @@
 "The line of text that does not match the Grok Expression will be 
appended to the last field of the prior message.");
 static final AllowableValue SKIP_LINE = new 
AllowableValue("skip-line", "Skip Line",
 "The line of text that does not match the Grok Expression will be 
skipped.");
+static final AllowableValue ROUTE_TO_FAILURE = new 
AllowableValue("route-to-failure", "Route to failure",
--- End diff --

Since it is up to the processor using the GrokReader to determine if 
something will be routed to failure, should we call this "Failure" or "Produce 
Error" or something indicating that reader is going to throw an exception? 


> GrokReader - Add a failure option to unmatch behavior options
> -
>
> Key: NIFI-4136
> URL: https://issues.apache.org/jira/browse/NIFI-4136
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> At the moment, when using the GrokReader, if a line does not match the grok 
> expression (and is not part of a stack trace), the line can be either ignored 
> (the line will be completely skipped) or  appended to the last field from the 
> previous line.
> In the case where appending is not desired and that data should not be 
> ignored/deleted, we should add the option to route the full flow file to the 
> failure relationship. This way the flow file could be treated in a different 
> way (for example with SplitText and ExtractGrok to isolate the incorrect 
> lines and re-route the correct lines back to the Record processors).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3503) Create a 'SplitCSV' processor

[Michael Moser (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068426#comment-16068426
 ] 

Michael Moser commented on NIFI-3503:
-

concur

> Create a 'SplitCSV' processor
> -
>
> Key: NIFI-3503
> URL: https://issues.apache.org/jira/browse/NIFI-3503
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Wesley L Lawrence
>Priority: Minor
>
> While the 'SplitText' processor helps break up newline separated records into 
> individual files, it's not uncommon to have CSV files where records span 
> multiple lines, and 'SplitText' isn't able or meant to handle this.
> Currently, one can replace, remove, or escape newline characters that exist 
> in a single CSV record by searching within quoted columns with 'ReplaceText', 
> before passing the data onto 'SplitText'. However, this may not work in all 
> cases, or could potentially remove the valid newline character at the end of 
> a CSV record, if all edge cases aren't properly covered with regex.
> Having a dedicated 'SplitCSV' processor will solve this problem, and be a 
> simpler approach for users.
> See the following [Apache NiFi user email 
> thread|https://mail-archives.apache.org/mod_mbox/nifi-users/201702.mbox/%3CCAFuL2BbgymFXwu5fRyd8pP-zu6WkToqPE2Ek7bkyBg0_-cknqQ%40mail.gmail.com%3E]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3232) Extend UI menus to allow cascading (menu item > subitem)

[Andrew Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068386#comment-16068386
 ] 

Andrew Lim commented on NIFI-3232:
--

[~mcgilman] looks to be actively working this.

> Extend UI menus to allow cascading (menu item > subitem)
> 
>
> Key: NIFI-3232
> URL: https://issues.apache.org/jira/browse/NIFI-3232
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Rob Moran
>
> With expanding functionality UI menu options are growing, making the size of 
> some quite large. Cascading menus will allow better information hierarchy to 
> improve the presentation of available user actions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4142) Implement a ValidateRecord Processor

[Mark Payne (JIRA)]

Mark Payne created NIFI-4142:


 Summary: Implement a ValidateRecord Processor
 Key: NIFI-4142
 URL: https://issues.apache.org/jira/browse/NIFI-4142
 Project: Apache NiFi
  Issue Type: New Feature
  Components: Extensions
Reporter: Mark Payne
Assignee: Mark Payne


We need a processor that is capable of validating that all Records in a 
FlowFile adhere to the proper schema.

The Processor should be configured with a Record Reader and should route each 
record to either 'valid' or 'invalid' based on whether or not the record 
adheres to the reader's schema. A record would be invalid in any of the 
following cases:

- Missing field that is required according to the schema
- Extra field that is not present in schema (it should be configurable whether 
or not this is a failure)
- Field requires coercion and strict type checking enabled (this should also be 
configurable)
- Field is invalid, such as the value "hello" when it should be an integer





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3503) Create a 'SplitCSV' processor

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068359#comment-16068359
 ] 

Mark Payne commented on NIFI-3503:
--

I believe we can now close this ticket, as the SplitRecord processor should now 
provide this capability. Any objections?

> Create a 'SplitCSV' processor
> -
>
> Key: NIFI-3503
> URL: https://issues.apache.org/jira/browse/NIFI-3503
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Wesley L Lawrence
>Priority: Minor
>
> While the 'SplitText' processor helps break up newline separated records into 
> individual files, it's not uncommon to have CSV files where records span 
> multiple lines, and 'SplitText' isn't able or meant to handle this.
> Currently, one can replace, remove, or escape newline characters that exist 
> in a single CSV record by searching within quoted columns with 'ReplaceText', 
> before passing the data onto 'SplitText'. However, this may not work in all 
> cases, or could potentially remove the valid newline character at the end of 
> a CSV record, if all edge cases aren't properly covered with regex.
> Having a dedicated 'SplitCSV' processor will solve this problem, and be a 
> simpler approach for users.
> See the following [Apache NiFi user email 
> thread|https://mail-archives.apache.org/mod_mbox/nifi-users/201702.mbox/%3CCAFuL2BbgymFXwu5fRyd8pP-zu6WkToqPE2Ek7bkyBg0_-cknqQ%40mail.gmail.com%3E]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-1716) Implement a SplitCsv processor, possibly also a GetCSV

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068358#comment-16068358
 ] 

Mark Payne commented on NIFI-1716:
--

I believe we can now close this ticket, as the SplitRecord processor should now 
provide this capability. Any objections?

> Implement a SplitCsv processor, possibly also a GetCSV
> --
>
> Key: NIFI-1716
> URL: https://issues.apache.org/jira/browse/NIFI-1716
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Reporter: Dmitry Goldenberg
>
> I'm proposing a SplitCSV processor dedicated specifically to splitting CSV 
> content which is assumed to be in the flowfile-content of its incoming 
> flowfiles.
> It appears that the current mode of splitting a CSV file is by using the 
> SplitText processor. However, it'd be great to have a CSV splitter to read 
> CSV records one by one and use the header row's header names to convert each 
> record into a FlowFile, with attributes set to correspond to the headers.
> Whether or not the first row is a header should be a boolean configuration 
> option.  In the absence of a header row, some sensible default column names 
> should be utilized, for example, one convention could be: column1, column2, 
> column3, etc. (or a naming strategy could be provided by the user in the 
> configuration).
> Another option on the splitter needs to be the delimiter character (defaulted 
> to comma).
> Empty lines shall be skipped from processing.
> Extracted cell values shall be (optionally) whitespace-trimmed.
> Jagged rows must have some sensible handling:
> 1) For a given row, if there are fewer cells than in the header row, cells 
> shall be assigned to columns left to right, and any missing cells shall be 
> considered empty.
> 2) For a given row, if there are more cells than in the header row, a 
> (non-fatal) error shall be generated for the row and the row shall be dropped 
> from processing.
> As typically done with CSV, delimiter characters are ignored within quotes.
> Elements may span multiple lines by having embedded carriage returns; such 
> elements must be quoted.
> NIFI-1280 asks for a way to specify which columns are to be kept or skipped. 
> I'm proposing that instead of a separate processor, this would be implemented 
> as a configuration option on SplitCSV (a list of 0-based indices of columns 
> that are to be kept).
> It may also make sense to expose a GetCSV ingress component which would share 
> most of its functionality with SplitCSV.  Perhaps it's easiest if users just 
> follow a GetFile with SplitCSV, however in some cases it makes sense to save 
> on reading the file into a flowfile-content but rather process all CSV data 
> in-place, within a GetCSV.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1891: NIFI-4008: Allow 0 or more records within a message...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1891#discussion_r124803429
  
--- Diff: 
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-0-10-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/ConsumerLease.java
 ---
@@ -418,82 +419,106 @@ private void writeDemarcatedData(final 
ProcessSession session, final List> records, final TopicPartition 
topicPartition) {
+private void writeRecordData(final ProcessSession session, final 
List> messages, final TopicPartition 
topicPartition) {
 // In order to obtain a RecordReader from the RecordReaderFactory, 
we need to give it a FlowFile.
 // We don't want to create a new FlowFile for each record that we 
receive, so we will just create
 // a "temporary flowfile" that will be removed in the finally 
block below and use that to pass to
 // the createRecordReader method.
 final FlowFile tempFlowFile = session.create();
 RecordSetWriter writer = null;
 
+final BiConsumer, Exception> 
handleParseFailure = (consumerRecord, e) -> {
--- End diff --

Unfortunately, I think this PR very much conflicts with NIFI-4046/PR 1906, 
which I just pushed another commit to, in order to address your feedback. 
Either PR could be merged first and then the other updated. We both refactored 
this piece of code to make it re-usable. However, with the new refactoring for 
4046, I think I prefer the method there, of having a separate method instead of 
a lambda because we end up passing in an optional message. Thoughts?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4008) ConsumeKafkaRecord_0_10 assumes there is always one Record in a message

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068355#comment-16068355
 ] 

ASF GitHub Bot commented on NIFI-4008:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1891#discussion_r124803429
  
--- Diff: 
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-0-10-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/ConsumerLease.java
 ---
@@ -418,82 +419,106 @@ private void writeDemarcatedData(final 
ProcessSession session, final List> records, final TopicPartition 
topicPartition) {
+private void writeRecordData(final ProcessSession session, final 
List> messages, final TopicPartition 
topicPartition) {
 // In order to obtain a RecordReader from the RecordReaderFactory, 
we need to give it a FlowFile.
 // We don't want to create a new FlowFile for each record that we 
receive, so we will just create
 // a "temporary flowfile" that will be removed in the finally 
block below and use that to pass to
 // the createRecordReader method.
 final FlowFile tempFlowFile = session.create();
 RecordSetWriter writer = null;
 
+final BiConsumer, Exception> 
handleParseFailure = (consumerRecord, e) -> {
--- End diff --

Unfortunately, I think this PR very much conflicts with NIFI-4046/PR 1906, 
which I just pushed another commit to, in order to address your feedback. 
Either PR could be merged first and then the other updated. We both refactored 
this piece of code to make it re-usable. However, with the new refactoring for 
4046, I think I prefer the method there, of having a separate method instead of 
a lambda because we end up passing in an optional message. Thoughts?


> ConsumeKafkaRecord_0_10 assumes there is always one Record in a message
> ---
>
> Key: NIFI-4008
> URL: https://issues.apache.org/jira/browse/NIFI-4008
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>
> ConsumeKafkaRecord_0_10 uses ConsumerLease underneath, and it [assumes there 
> is one Record available in a consumed 
> message|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-0-10-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/ConsumerLease.java#L434]
>  retrieved from a Kafka topic.
> But in fact, a message can contain 0 or more records in it. For example, with 
> a record schema shown below:
> {code}
> {
>   "type": "record",
>   "name": "temp",
>   "fields" : [
> {"name": "value", "type": "string"}
>   ]
> }
> {code}
> Multiple records can be sent within a single message, e.g. using JSON:
> {code}
> [{"value": "a"}, {"value": "b"}, {"value": "c"}]
> {code}
> But ConsumeKafkaRecord only outputs the first record:
> {code}
> [{"value": "a"}]
> {code}
> Also, if a message doesn't contain any record in it, the processor fails with 
> NullPointerException.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4046) ConsumeKafkaRecord can throw NPE if using an Avro Reader and data on topic is missing a required field

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068350#comment-16068350
 ] 

ASF GitHub Bot commented on NIFI-4046:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1906
  
@ijokarumawak that's a great catch! Pushed a new commit to address. Thanks!


> ConsumeKafkaRecord can throw NPE if using an Avro Reader and data on topic is 
> missing a required field
> --
>
> Key: NIFI-4046
> URL: https://issues.apache.org/jira/browse/NIFI-4046
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> I am using PublishKafkaRecord to push some data to a Kafka topic using the 
> Avro writer. I not embedding the Avro schema in the content but rather using 
> a Schema Registry.
> I then have a ConsumeKafkaRecord that pulls that Avro data, but is using a 
> Schema that has an extra field, which is not present in the data. As a 
> result, I get a NullPointerException when trying to pull data from Kafka:
> {code}
> 2017-06-08 14:54:12,499 ERROR [Timer-Driven Process Thread-22] 
> o.a.n.p.k.pubsub.ConsumeKafkaRecord_0_10 
> ConsumeKafkaRecord_0_10[id=84480f03-015c-1000-4aab-9a035812bb86] Exception 
> while processing data from kafka so will close the lease 
> org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@6c39d011
>  due to org.apache.nifi.processor.exception.ProcessException: 
> java.lang.NullPointerException: 
> org.apache.nifi.processor.exception.ProcessException: 
> java.lang.NullPointerException
> org.apache.nifi.processor.exception.ProcessException: 
> java.lang.NullPointerException
> at 
> org.apache.nifi.processors.kafka.pubsub.ConsumerLease.writeRecordData(ConsumerLease.java:514)
> at 
> org.apache.nifi.processors.kafka.pubsub.ConsumerLease.lambda$processRecords$2(ConsumerLease.java:320)
> at 
> java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1540)
> at 
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
> at 
> org.apache.nifi.processors.kafka.pubsub.ConsumerLease.processRecords(ConsumerLease.java:307)
> at 
> org.apache.nifi.processors.kafka.pubsub.ConsumerLease.poll(ConsumerLease.java:168)
> at 
> org.apache.nifi.processors.kafka.pubsub.ConsumeKafkaRecord_0_10.onTrigger(ConsumeKafkaRecord_0_10.java:327)
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException: null
> at 
> org.apache.nifi.processors.kafka.pubsub.ConsumerLease.writeRecordData(ConsumerLease.java:458)
> ... 18 common frames omitted
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1906: NIFI-4046: If we are unable to parse out any records from ...

[markap14]

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1906
  
@ijokarumawak that's a great catch! Pushed a new commit to address. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4118) Create Nifi RethinkDB Put processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068322#comment-16068322
 ] 

ASF GitHub Bot commented on NIFI-4118:
--

Github user mans2singh commented on the issue:

https://github.com/apache/nifi/pull/1942
  
Hey Folks:

Can you please give me your feedback on this RethinkDB Put processor ?

Thanks

Mans


> Create Nifi RethinkDB Put processor
> ---
>
> Key: NIFI-4118
> URL: https://issues.apache.org/jira/browse/NIFI-4118
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.3.0
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: document, stream,
> Fix For: 1.4.0
>
>
> Create Nifi processor for streaming documents into RethinkDB.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1942: NIFI-4118 First commit of RethinkDB put processor

[mans2singh]

Github user mans2singh commented on the issue:

https://github.com/apache/nifi/pull/1942
  
Hey Folks:

Can you please give me your feedback on this RethinkDB Put processor ?

Thanks

Mans


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-4060) Create a MergeRecord Processor

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4060:
-
Fix Version/s: 1.4.0
   Status: Patch Available  (was: Open)

> Create a MergeRecord Processor
> --
>
> Key: NIFI-4060
> URL: https://issues.apache.org/jira/browse/NIFI-4060
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> When record-oriented data is received one record or a time or needs to be 
> split into small chunks for one reason or another, it will be helpful to be 
> able to combine those records into a single FlowFile that is made up of many 
> records for efficiency purposes, or to deliver to downstream systems as 
> larger batches. This processor should function similarly to MergeContent but 
> make use of Record Readers and Record Writer so that users don't have to deal 
> with headers, footers, demarcators, etc.
> The Processor will also need to ensure that records only get merge into the 
> same FlowFile if they have compatible schemas.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4060) Create a MergeRecord Processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068256#comment-16068256
 ] 

ASF GitHub Bot commented on NIFI-4060:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/1958

NIFI-4060: Initial implementation of MergeRecord

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4060

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1958.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1958


commit 20e86cbdc8d32cbe088a41595e04e84e70ddd503
Author: Mark Payne 
Date:   2017-06-26T17:15:03Z

NIFI-4060: Initial implementation of MergeRecord




> Create a MergeRecord Processor
> --
>
> Key: NIFI-4060
> URL: https://issues.apache.org/jira/browse/NIFI-4060
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> When record-oriented data is received one record or a time or needs to be 
> split into small chunks for one reason or another, it will be helpful to be 
> able to combine those records into a single FlowFile that is made up of many 
> records for efficiency purposes, or to deliver to downstream systems as 
> larger batches. This processor should function similarly to MergeContent but 
> make use of Record Readers and Record Writer so that users don't have to deal 
> with headers, footers, demarcators, etc.
> The Processor will also need to ensure that records only get merge into the 
> same FlowFile if they have compatible schemas.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1958: NIFI-4060: Initial implementation of MergeRecord

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/1958

NIFI-4060: Initial implementation of MergeRecord

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4060

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1958.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1958


commit 20e86cbdc8d32cbe088a41595e04e84e70ddd503
Author: Mark Payne 
Date:   2017-06-26T17:15:03Z

NIFI-4060: Initial implementation of MergeRecord




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-4086) Docker image produced by Dockerfile is larger than needed

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4086:
-
Issue Type: Improvement  (was: Bug)

> Docker image produced by Dockerfile is larger than needed
> -
>
> Key: NIFI-4086
> URL: https://issues.apache.org/jira/browse/NIFI-4086
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Docker
>Reporter: Niels Zeilemaker
>
> The Dockerfile has a chown action after the curl, which more or less doubles 
> the size of the resulting docker image. Merging the chown step into the curl 
> step fixes this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4122) GetMongo should be able to group results into a set of flowfiles

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4122:
-
Component/s: Extensions

> GetMongo should be able to group results into a set of flowfiles
> 
>
> Key: NIFI-4122
> URL: https://issues.apache.org/jira/browse/NIFI-4122
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Mike Thomsen
>Priority: Minor
>  Labels: getmongo, mongodb, nifi
> Fix For: 1.4.0
>
>
> GetMongo should be able to take a user-defined limit and group results by 
> that size into flowfiles rather than having only the ability to do a 1:1 
> relationship between result and flowfile.
> For example, if the user specifies 100, 100 results should be grouped 
> together and turned into a JSON array that can be broken up later as needed.
> This need arose when doing a bulk data ingestion from Mongo. We had shy of 
> 400k documents, and the 1:1 generation of flowfiles blew right through our 
> limits on the content repository. Adding this feature would make it feasible 
> to control that sort of behavior more thoroughly for events like bulk 
> ingestion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-4122) GetMongo should be able to group results into a set of flowfiles

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-4122.
--
   Resolution: Fixed
Fix Version/s: 1.4.0

> GetMongo should be able to group results into a set of flowfiles
> 
>
> Key: NIFI-4122
> URL: https://issues.apache.org/jira/browse/NIFI-4122
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Mike Thomsen
>Priority: Minor
>  Labels: getmongo, mongodb, nifi
> Fix For: 1.4.0
>
>
> GetMongo should be able to take a user-defined limit and group results by 
> that size into flowfiles rather than having only the ability to do a 1:1 
> relationship between result and flowfile.
> For example, if the user specifies 100, 100 results should be grouped 
> together and turned into a JSON array that can be broken up later as needed.
> This need arose when doing a bulk data ingestion from Mongo. We had shy of 
> 400k documents, and the 1:1 generation of flowfiles blew right through our 
> limits on the content repository. Adding this feature would make it feasible 
> to control that sort of behavior more thoroughly for events like bulk 
> ingestion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4122) GetMongo should be able to group results into a set of flowfiles

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068155#comment-16068155
 ] 

ASF subversion and git services commented on NIFI-4122:
---

Commit 51727974485ffb9617c8d510377e70ea7b50cb53 in nifi's branch 
refs/heads/master from [~mike.thomsen]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=5172797 ]

NIFI-4122 Added the ability to combine multiple Mongo result documents into a 
single output JSON array.

Signed-off-by: Pierre Villard 

This closes #1948.


> GetMongo should be able to group results into a set of flowfiles
> 
>
> Key: NIFI-4122
> URL: https://issues.apache.org/jira/browse/NIFI-4122
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>  Labels: getmongo, mongodb, nifi
>
> GetMongo should be able to take a user-defined limit and group results by 
> that size into flowfiles rather than having only the ability to do a 1:1 
> relationship between result and flowfile.
> For example, if the user specifies 100, 100 results should be grouped 
> together and turned into a JSON array that can be broken up later as needed.
> This need arose when doing a bulk data ingestion from Mongo. We had shy of 
> 400k documents, and the 1:1 generation of flowfiles blew right through our 
> limits on the content repository. Adding this feature would make it feasible 
> to control that sort of behavior more thoroughly for events like bulk 
> ingestion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4122) GetMongo should be able to group results into a set of flowfiles

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068156#comment-16068156
 ] 

ASF GitHub Bot commented on NIFI-4122:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1948


> GetMongo should be able to group results into a set of flowfiles
> 
>
> Key: NIFI-4122
> URL: https://issues.apache.org/jira/browse/NIFI-4122
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>  Labels: getmongo, mongodb, nifi
>
> GetMongo should be able to take a user-defined limit and group results by 
> that size into flowfiles rather than having only the ability to do a 1:1 
> relationship between result and flowfile.
> For example, if the user specifies 100, 100 results should be grouped 
> together and turned into a JSON array that can be broken up later as needed.
> This need arose when doing a bulk data ingestion from Mongo. We had shy of 
> 400k documents, and the 1:1 generation of flowfiles blew right through our 
> limits on the content repository. Adding this feature would make it feasible 
> to control that sort of behavior more thoroughly for events like bulk 
> ingestion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1948: NIFI-4122 Added the ability to combine multiple Mon...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1948


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4122) GetMongo should be able to group results into a set of flowfiles

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068146#comment-16068146
 ] 

ASF GitHub Bot commented on NIFI-4122:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/1948
  
+1, thanks @MikeThomsen 
I'll revert the change on ``@Ignore`` while merging to master (it's needed 
to keep a valid build, this test class being more an integration test).


> GetMongo should be able to group results into a set of flowfiles
> 
>
> Key: NIFI-4122
> URL: https://issues.apache.org/jira/browse/NIFI-4122
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>  Labels: getmongo, mongodb, nifi
>
> GetMongo should be able to take a user-defined limit and group results by 
> that size into flowfiles rather than having only the ability to do a 1:1 
> relationship between result and flowfile.
> For example, if the user specifies 100, 100 results should be grouped 
> together and turned into a JSON array that can be broken up later as needed.
> This need arose when doing a bulk data ingestion from Mongo. We had shy of 
> 400k documents, and the 1:1 generation of flowfiles blew right through our 
> limits on the content repository. Adding this feature would make it feasible 
> to control that sort of behavior more thoroughly for events like bulk 
> ingestion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1948: NIFI-4122 Added the ability to combine multiple Mongo resu...

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/1948
  
+1, thanks @MikeThomsen 
I'll revert the change on ``@Ignore`` while merging to master (it's needed 
to keep a valid build, this test class being more an integration test).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1937: NIFI-4105 support the specified Maximum value colum...

[ggthename]

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124765825
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
 ---
@@ -75,19 +81,64 @@
 @Tags({"cassandra", "cql", "select"})
 @EventDriven
 @InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
-@CapabilityDescription("Execute provided Cassandra Query Language (CQL) 
select query on a Cassandra 1.x, 2.x, or 3.0.x cluster. Query result "
-+ "may be converted to Avro or JSON format. Streaming is used so 
arbitrarily large result sets are supported. This processor can be "
+@CapabilityDescription("Executes provided Cassandra Query Language (CQL) 
select query on a Cassandra to fetch all rows whose values"
++ "in the specified Maximum Value column(s) are larger than the 
previously-seen maxima.Query result"
++ "may be converted to Avro, JSON or CSV format. Streaming is used 
so arbitrarily large result sets are supported. This processor can be "
 + "scheduled to run on a timer, or cron expression, using the 
standard scheduling methods, or it can be triggered by an incoming FlowFile. "
 + "If it is triggered by an incoming FlowFile, then attributes of 
that FlowFile will be available when evaluating the "
 + "select query. FlowFile attribute 'executecql.row.count' 
indicates how many rows were selected.")
+@Stateful(scopes = Scope.CLUSTER, description = "After performing query, 
the maximum value of the specified column is stored, "
++ "fetch all rows whose values in the specified Maximum Value 
column(s) are larger than the previously-seen maximum"
++ "State is stored across the cluster so that the next time this 
Processor can be run with min and max values")
 @WritesAttributes({@WritesAttribute(attribute = "executecql.row.count", 
description = "The number of rows returned by the CQL query")})
 public class QueryCassandra extends AbstractCassandraProcessor {
 
+public static final String CSV_FORMAT = "CSV";
 public static final String AVRO_FORMAT = "Avro";
 public static final String JSON_FORMAT = "JSON";
 
+public static final String CASSANDRA_WATERMARK_MIN_VALUE_ID = 
"CASSANDRA_WATERMARK_MIN_VALUE_ID";
+public static final String CASSANDRA_WATERMARK_MAX_VALUE_ID = 
"CASSANDRA_WATERMARK_MAX_VALUE_ID";
+
 public static final String RESULT_ROW_COUNT = "executecql.row.count";
 
+public static final PropertyDescriptor INIT_WATERMARK = new 
PropertyDescriptor.Builder().name("Initial Watermark Value")
+.description("Use it only once.")
--- End diff --

I'm not familiar with english. But I'll try again!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (NIFI-4086) Docker image produced by Dockerfile is larger than needed

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-4086.
--
Resolution: Duplicate

> Docker image produced by Dockerfile is larger than needed
> -
>
> Key: NIFI-4086
> URL: https://issues.apache.org/jira/browse/NIFI-4086
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Docker
>Reporter: Niels Zeilemaker
>
> The Dockerfile has a chown action after the curl, which more or less doubles 
> the size of the resulting docker image. Merging the chown step into the curl 
> step fixes this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4105) support the specified Maximum value column and CSV Stream for Cassandra

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068138#comment-16068138
 ] 

ASF GitHub Bot commented on NIFI-4105:
--

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124765427
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
 ---
@@ -75,19 +81,64 @@
 @Tags({"cassandra", "cql", "select"})
 @EventDriven
 @InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
-@CapabilityDescription("Execute provided Cassandra Query Language (CQL) 
select query on a Cassandra 1.x, 2.x, or 3.0.x cluster. Query result "
-+ "may be converted to Avro or JSON format. Streaming is used so 
arbitrarily large result sets are supported. This processor can be "
+@CapabilityDescription("Executes provided Cassandra Query Language (CQL) 
select query on a Cassandra to fetch all rows whose values"
--- End diff --

Thank you for your wording! I really appreciate it.

I think it is a little different from GenerateTableFetch. Because 
GenerateTableFetch use various kinds of DatabaseAdapter for RDB. They usually 
use "ORDER BY" keyword.

But Cassandra does not have this feature. And It has specialized structure 
based on Column Family, not Row

So I built this processor with range query.


> support the specified Maximum value column and CSV Stream for Cassandra
> ---
>
> Key: NIFI-4105
> URL: https://issues.apache.org/jira/browse/NIFI-4105
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.3.0
>Reporter: Yoonwon Ko
>
> I'm trying to find a CassandraProcessor to fetch rows whose values in the 
> specified Maximum Value columns are larger than the previously-seen maximum 
> like QueryDatabaseTable.
> But I found only QueryCassandra. It just executes same CQL everytime without 
> keeping maximum value.
> and I think we also need convertToCsvStream option.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #1937: NIFI-4105 support the specified Maximum value colum...

[ggthename]

Github user ggthename commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1937#discussion_r124765427
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
 ---
@@ -75,19 +81,64 @@
 @Tags({"cassandra", "cql", "select"})
 @EventDriven
 @InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
-@CapabilityDescription("Execute provided Cassandra Query Language (CQL) 
select query on a Cassandra 1.x, 2.x, or 3.0.x cluster. Query result "
-+ "may be converted to Avro or JSON format. Streaming is used so 
arbitrarily large result sets are supported. This processor can be "
+@CapabilityDescription("Executes provided Cassandra Query Language (CQL) 
select query on a Cassandra to fetch all rows whose values"
--- End diff --

Thank you for your wording! I really appreciate it.

I think it is a little different from GenerateTableFetch. Because 
GenerateTableFetch use various kinds of DatabaseAdapter for RDB. They usually 
use "ORDER BY" keyword.

But Cassandra does not have this feature. And It has specialized structure 
based on Column Family, not Row

So I built this processor with range query.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4086) Docker image produced by Dockerfile is larger than needed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068131#comment-16068131
 ] 

ASF GitHub Bot commented on NIFI-4086:
--

Github user NielsZeilemaker closed the pull request at:

https://github.com/apache/nifi/pull/1926


> Docker image produced by Dockerfile is larger than needed
> -
>
> Key: NIFI-4086
> URL: https://issues.apache.org/jira/browse/NIFI-4086
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Docker
>Reporter: Niels Zeilemaker
>
> The Dockerfile has a chown action after the curl, which more or less doubles 
> the size of the resulting docker image. Merging the chown step into the curl 
> step fixes this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)