[jira] [Commented] (NIFI-4838) Make GetMongo support multiple commits and give some progress indication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362286#comment-16362286
 ] 

ASF GitHub Bot commented on NIFI-4838:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2448
  
@mattyb149 Could you take a look at this? It's modeled on your work on 
ExecuteSQL.


> Make GetMongo support multiple commits and give some progress indication
> 
>
> Key: NIFI-4838
> URL: https://issues.apache.org/jira/browse/NIFI-4838
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> It shouldn't wait until the end to do a commit() call because the effect is 
> that GetMongo looks like it has hung to a user who is pulling a very large 
> data set.
> It should also have an option for running a count query to get the current 
> approximate count of documents that would match the query and append an 
> attribute that indicates where a flowfile stands in the total result count. 
> Ex:
> query.progress.point.start = 2500
> query.progress.point.end = 5000
> query.count.estimate = 17,568,231



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2448: NIFI-4838 Added configurable progressive commits to GetMon...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2448
  
@mattyb149 Could you take a look at this? It's modeled on your work on 
ExecuteSQL.


---

[jira] [Commented] (NIFI-4836) Allow QueryDatabaseTables to send out batches of flow files while result set is being processed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362285#comment-16362285
 ] 

ASF GitHub Bot commented on NIFI-4836:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2447
  
@pvillard31 @joewitt @alopresto LGTM and should be an easy win to merge.


> Allow QueryDatabaseTables to send out batches of flow files while result set 
> is being processed
> ---
>
> Key: NIFI-4836
> URL: https://issues.apache.org/jira/browse/NIFI-4836
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Currently QueryDatabaseTable (QDT) will not transfer the outgoing flowfiles 
> to the downstream relationship(s) until the entire result set has been 
> processed (regardless of whether Max Rows Per Flow File is set). This is so 
> the maxvalue.* and fragment.count attributes can be set correctly for each 
> flow file.
> However for very large result sets, the initial fetch can take a long time, 
> and depending on the setting of Max Rows Per FlowFile, there could be a great 
> number of FlowFiles transferred downstream as a large burst at the end of QDT 
> execution.
> It would be nice for the user to be able to choose to have FlowFiles be 
> transferred downstream while the result set is still being processed. This 
> alleviates the "large burst at the end" by replacing it with smaller output 
> batches during processing. The tradeoff will be that if an Output Batch Size 
> is set, then the maxvalue.* and fragment.count attributes will not be set on 
> the outgoing flow files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2447: NIFI-4836: Allow output of FlowFiles during result set pro...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2447
  
@pvillard31 @joewitt @alopresto LGTM and should be an easy win to merge.


---

[jira] [Commented] (NIFI-4164) Realistic Time Series Processor Simulator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362294#comment-16362294
 ] 

ASF GitHub Bot commented on NIFI-4164:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/1997
  
@cherrera2001 I haven't seen any activity here, but before we go for round 
two I would recommend reading the thread "Will you accept contributions in 
Scala" on nifi-dev. If you're not subscribed, go 
[here](https://mail-archives.apache.org/mod_mbox/nifi-dev/201802.mbox/browser). 
Most of the committers seem to be against contributions in Scala so that small 
Scala file should be rewritten in Java. (As a Groovy fan I can commiserate on 
not getting to use one's preferred language)


> Realistic Time Series Processor Simulator
> -
>
> Key: NIFI-4164
> URL: https://issues.apache.org/jira/browse/NIFI-4164
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Chris Herrera
>Assignee: Chris Herrera
>Priority: Minor
>  Labels: features
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> In order to validate several flows that deal with sensor data, it would be 
> good to have a built in time series simulator processor that generates data 
> and can send it out via a flow file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362292#comment-16362292
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167858912
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
+}
+)
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
+static final AllowableValue ROW_ID_BODY = new AllowableValue("body", 
"FlowFile content", "Get the row key(s) from the flowfile content.");
--- End diff --

I'd suggest keep the naming consistent. Both of `body` and `content` are 
used, but it would be cleaner if we use just one of them. If so, I'd prefer 
`content`.


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362288#comment-16362288
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167851239
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
+}
+)
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
--- End diff --

`@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)` should be 
added.


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362291#comment-16362291
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167854208
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
--- End diff --

It says "Only written when a flowfile contains multiple deletes." but the 
attribute is written when a FlowFile only contains single row id. It would be 
more accurate if we say "Only written when deleting Row IDs from flowfile 
content."


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362293#comment-16362293
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167857406
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
+}
+)
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
+static final AllowableValue ROW_ID_BODY = new AllowableValue("body", 
"FlowFile content", "Get the row key(s) from the flowfile content.");
+static final AllowableValue ROW_ID_ATTR = new AllowableValue("attr", 
"FlowFile attributes", "Get the row key from an expression language 
statement.");
+
+static final String RESTART_INDEX = "restart.index";
+static final String ROWKEY_START = "rowkey.start";
+static final String ROWKEY_END   = "rowkey.end";
+
+static final PropertyDescriptor ROW_ID_LOCATION = new 
PropertyDescriptor.Builder()
+.name("delete-hb-row-id-location")
+.displayName("Row ID Location")
+.description("The location of the row ID to use for building 
the delete. Can be from the content or an expression language statement.")
+.required(true)
+.defaultValue(ROW_ID_BODY.getValue())
+.allowableValues(ROW_ID_BODY, ROW_ID_ATTR)
+.addValidator(Validator.VALID)
+.build();
+
+static final PropertyDescriptor FLOWFILE_FETCH_COUNT = new 
PropertyDescriptor.Builder()
+.name("delete-hb-flowfile-fetch-count")
+.displayName("Flowfile Fetch Count")
+.description("The number of flowfiles to fetch per run.")
+.required(true)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.defaultValue("5")
+.expressionLanguageSupported(false)
+.build();
+
+static final PropertyDescriptor BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("delete-hb-batch-size")
+  

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362289#comment-16362289
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167851490
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/test/java/org/apache/nifi/hbase/TestDeleteHBaseRow.java
 ---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+public class TestDeleteHBaseRow {
+private TestRunner runner;
+private MockHBaseClientService hBaseClient;
+
+@Before
+public void setup() throws InitializationException {
+runner = TestRunners.newTestRunner(new DeleteHBaseRow());
+
+hBaseClient = new MockHBaseClientService();
+runner.addControllerService("hbaseClient", hBaseClient);
+runner.enableControllerService(hBaseClient);
+
+runner.setProperty(DeleteHBaseRow.TABLE_NAME, "nifi");
+runner.setProperty(DeleteHBaseRow.HBASE_CLIENT_SERVICE, 
"hbaseClient");
+}
+
+List populateTable(int max) {
+List ids = new ArrayList<>();
+for (int index = 0; index < max; index++) {
+String uuid = UUID.randomUUID().toString();
+ids.add(uuid);
+Map cells = new HashMap<>();
+cells.put("test", UUID.randomUUID().toString());
+hBaseClient.addResult(uuid, cells, System.currentTimeMillis());
+}
+
+return ids;
+}
+
+@Test
+public void testSimpleDelete() {
+List ids = populateTable(100);
+
+runner.setProperty(DeleteHBaseRow.BATCH_SIZE, "100");
+runner.setProperty(DeleteHBaseRow.FLOWFILE_FETCH_COUNT, "100");
+for (String id : ids) {
+runner.enqueue(id);
+}
+
+runner.run(1, true);
+Assert.assertTrue("The mock client was not empty.", 
hBaseClient.isEmpty());
+}
+
+private String buildSeparatedString(List ids, String 
separator) {
+StringBuilder sb = new StringBuilder();
+for (int index = 1; index <= ids.size(); index++) {
+sb.append(ids.get(index - 1)).append(separator);
+}
+
+return sb.toString();
+}
+
+private void testSeparatedDeletes(String separator) {
+testSeparatedDeletes(separator, separator, new HashMap());
+}
+
+private void testSeparatedDeletes(String separator, String 
separatorProp, Map attrs) {
+List ids = populateTable(1);
+runner.setProperty(DeleteHBaseRow.KEY_SEPARATOR, separator);
+runner.setProperty(DeleteHBaseRow.BATCH_SIZE, "100");
+runner.enqueue(buildSeparatedString(ids, separatorProp), attrs);
+runner.run(1, true);
+
+Assert.assertTrue("The mock client was not empty.", 
hBaseClient.isEmpty());
+}
+
+@Test
+public void testDeletesSeparatedByNewLines() {
+testSeparatedDeletes("\n");
+}
+
+@Test
+public void testDeletesSeparatedByCommas() {
+testSeparatedDeletes(",");
+}
+
+@Test
+public void testDeleteWithELSeparator() {
+runner.setValidateExpressionUsage(true);
+Map attrs = new HashMap<>();
+

[GitHub] nifi issue #1997: NIFI-4164 Adding a realistic time simulator processor to N...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/1997
  
@cherrera2001 I haven't seen any activity here, but before we go for round 
two I would recommend reading the thread "Will you accept contributions in 
Scala" on nifi-dev. If you're not subscribed, go 
[here](https://mail-archives.apache.org/mod_mbox/nifi-dev/201802.mbox/browser). 
Most of the committers seem to be against contributions in Scala so that small 
Scala file should be rewritten in Java. (As a Groovy fan I can commiserate on 
not getting to use one's preferred language)


---

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167857406
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
+}
+)
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
+static final AllowableValue ROW_ID_BODY = new AllowableValue("body", 
"FlowFile content", "Get the row key(s) from the flowfile content.");
+static final AllowableValue ROW_ID_ATTR = new AllowableValue("attr", 
"FlowFile attributes", "Get the row key from an expression language 
statement.");
+
+static final String RESTART_INDEX = "restart.index";
+static final String ROWKEY_START = "rowkey.start";
+static final String ROWKEY_END   = "rowkey.end";
+
+static final PropertyDescriptor ROW_ID_LOCATION = new 
PropertyDescriptor.Builder()
+.name("delete-hb-row-id-location")
+.displayName("Row ID Location")
+.description("The location of the row ID to use for building 
the delete. Can be from the content or an expression language statement.")
+.required(true)
+.defaultValue(ROW_ID_BODY.getValue())
+.allowableValues(ROW_ID_BODY, ROW_ID_ATTR)
+.addValidator(Validator.VALID)
+.build();
+
+static final PropertyDescriptor FLOWFILE_FETCH_COUNT = new 
PropertyDescriptor.Builder()
+.name("delete-hb-flowfile-fetch-count")
+.displayName("Flowfile Fetch Count")
+.description("The number of flowfiles to fetch per run.")
+.required(true)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.defaultValue("5")
+.expressionLanguageSupported(false)
+.build();
+
+static final PropertyDescriptor BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("delete-hb-batch-size")
+.displayName("Batch Size")
+.description("The number of deletes to send per batch.")
+.required(true)
+.defaultValue("50")
+

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362290#comment-16362290
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167854611
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
--- End diff --

Same comment as "rowkey.start" on the description applied here.


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167854208
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
--- End diff --

It says "Only written when a flowfile contains multiple deletes." but the 
attribute is written when a FlowFile only contains single row id. It would be 
more accurate if we say "Only written when deleting Row IDs from flowfile 
content."


---

[GitHub] nifi issue #1997: NIFI-4164 Adding a realistic time simulator processor to N...

[cherrera2001]

Github user cherrera2001 commented on the issue:

https://github.com/apache/nifi/pull/1997
  
Thanks Mike I did see it, and as such it is getting re written :)


---

[jira] [Commented] (NIFI-4875) Import flow from nifi-registry on bootstrap/initial start

[Daniel Oakley (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362150#comment-16362150
 ] 

Daniel Oakley commented on NIFI-4875:
-

Hi [~pvillard] , I think it would be good in the standard build. I am using a 
custom docker image anyway.

I would still hope after initializing the flow.xml.gz the Nifi controller is 
aware the flow was sourced from the nifi-registry, so that updates can be 
detected and applied as per normal operation when nifi is integrated with the 
registry.

Another issue that this relates to, which I am not sure is nifi or registry 
related, is that nifi seems only to store a process group in the registry, not 
an entire flow. So it is not clear to me how a new Nifi could be initialized 
with a flow stored in the registry without first creating a new flow, and then 
adding a process group that loads a stored registry flow into it. Or can a 
process group saved to the registry be extracted as a new top-level flow that 
doesn't require the manual addition of a process group? I find that requirement 
somewhat strange and confusing.

> Import flow from nifi-registry on bootstrap/initial start
> -
>
> Key: NIFI-4875
> URL: https://issues.apache.org/jira/browse/NIFI-4875
> Project: Apache NiFi
>  Issue Type: Wish
>  Components: Flow Versioning
>Affects Versions: 1.5.0
>Reporter: Daniel Oakley
>Priority: Major
>  Labels: registry
>
> As I user I would like to be able to automate the deployment of a new Nifi 
> instance (e.g. using Docker, Ansible), and have that instance load a specific 
> versioned flow from the nifi-registry and start running it automatically.
> To enable this I would like to be able to specify the nifi-registry URL in 
> nifi properties file, and the flow to load/start via either the properties 
> file or a command line option.
> This will greatly increase the ease of deploying nifi in automated 
> environments such as Kubernetes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4872) NIFI component high resource usage annotation

[Jeff Storck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Storck updated NIFI-4872:
--
Description: 
NiFi Processors currently have no means to relay whether or not they have may 
be resource intensive or not. The idea here would be to introduce an Annotation 
that can be added to Processors that indicate they may cause high memory, disk, 
CPU, or network usage. For instance, any Processor that reads the FlowFile 
contents into memory (like many XML Processors for instance) may cause high 
memory usage. What ultimately determines if there is high 
memory/disk/cpu/network usage will depend on the FlowFiles being processed. 
With many of these components in the dataflow, it increases the risk of 
OutOfMemoryErrors and performance degradation.

The annotation should support one value from a fixed list of: CPU, Disk, 
Memory, Network.  It should also allow the developer to provide a custom 
description of the scenario that the component would fall under the high usage 
category.  The annotation should be able to be specified multiple times, for as 
many resources as it has the potential to be high usage.

By marking components with this new Annotation, we can update the generated 
Processor documentation to include this fact.

  was:
NiFi Processors currently have no means to relay whether or not they have may 
be resource intensive or not. The idea here would be to introduce an Annotation 
that can be added to Processors that indicate they may cause high memory, disk, 
CPU, or network usage. For instance, any Processor that reads the FlowFile 
contents into memory (like many XML Processors for instance) may cause high 
memory usage. What ultimately determines if there is high 
memory/disk/cpu/network usage will depend on the FlowFiles being processed. 
With many of these components in the dataflow, it increases the risk of 
OutOfMemoryErrors and performance degradation.

The annotation should support one or more values from a fixed list of: CPU, 
Disk, Memory, Network.

By marking components with this new Annotation, we can update the generated 
Processor documentation to include this fact.


> NIFI component high resource usage annotation
> -
>
> Key: NIFI-4872
> URL: https://issues.apache.org/jira/browse/NIFI-4872
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework, Core UI
>Affects Versions: 1.5.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Critical
>
> NiFi Processors currently have no means to relay whether or not they have may 
> be resource intensive or not. The idea here would be to introduce an 
> Annotation that can be added to Processors that indicate they may cause high 
> memory, disk, CPU, or network usage. For instance, any Processor that reads 
> the FlowFile contents into memory (like many XML Processors for instance) may 
> cause high memory usage. What ultimately determines if there is high 
> memory/disk/cpu/network usage will depend on the FlowFiles being processed. 
> With many of these components in the dataflow, it increases the risk of 
> OutOfMemoryErrors and performance degradation.
> The annotation should support one value from a fixed list of: CPU, Disk, 
> Memory, Network.  It should also allow the developer to provide a custom 
> description of the scenario that the component would fall under the high 
> usage category.  The annotation should be able to be specified multiple 
> times, for as many resources as it has the potential to be high usage.
> By marking components with this new Annotation, we can update the generated 
> Processor documentation to include this fact.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4164) Realistic Time Series Processor Simulator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362298#comment-16362298
 ] 

ASF GitHub Bot commented on NIFI-4164:
--

Github user cherrera2001 commented on the issue:

https://github.com/apache/nifi/pull/1997
  
Thanks Mike I did see it, and as such it is getting re written :)


> Realistic Time Series Processor Simulator
> -
>
> Key: NIFI-4164
> URL: https://issues.apache.org/jira/browse/NIFI-4164
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Chris Herrera
>Assignee: Chris Herrera
>Priority: Minor
>  Labels: features
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> In order to validate several flows that deal with sensor data, it would be 
> good to have a built in time series simulator processor that generates data 
> and can send it out via a flow file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167851490
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/test/java/org/apache/nifi/hbase/TestDeleteHBaseRow.java
 ---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+public class TestDeleteHBaseRow {
+private TestRunner runner;
+private MockHBaseClientService hBaseClient;
+
+@Before
+public void setup() throws InitializationException {
+runner = TestRunners.newTestRunner(new DeleteHBaseRow());
+
+hBaseClient = new MockHBaseClientService();
+runner.addControllerService("hbaseClient", hBaseClient);
+runner.enableControllerService(hBaseClient);
+
+runner.setProperty(DeleteHBaseRow.TABLE_NAME, "nifi");
+runner.setProperty(DeleteHBaseRow.HBASE_CLIENT_SERVICE, 
"hbaseClient");
+}
+
+List populateTable(int max) {
+List ids = new ArrayList<>();
+for (int index = 0; index < max; index++) {
+String uuid = UUID.randomUUID().toString();
+ids.add(uuid);
+Map cells = new HashMap<>();
+cells.put("test", UUID.randomUUID().toString());
+hBaseClient.addResult(uuid, cells, System.currentTimeMillis());
+}
+
+return ids;
+}
+
+@Test
+public void testSimpleDelete() {
+List ids = populateTable(100);
+
+runner.setProperty(DeleteHBaseRow.BATCH_SIZE, "100");
+runner.setProperty(DeleteHBaseRow.FLOWFILE_FETCH_COUNT, "100");
+for (String id : ids) {
+runner.enqueue(id);
+}
+
+runner.run(1, true);
+Assert.assertTrue("The mock client was not empty.", 
hBaseClient.isEmpty());
+}
+
+private String buildSeparatedString(List ids, String 
separator) {
+StringBuilder sb = new StringBuilder();
+for (int index = 1; index <= ids.size(); index++) {
+sb.append(ids.get(index - 1)).append(separator);
+}
+
+return sb.toString();
+}
+
+private void testSeparatedDeletes(String separator) {
+testSeparatedDeletes(separator, separator, new HashMap());
+}
+
+private void testSeparatedDeletes(String separator, String 
separatorProp, Map attrs) {
+List ids = populateTable(1);
+runner.setProperty(DeleteHBaseRow.KEY_SEPARATOR, separator);
+runner.setProperty(DeleteHBaseRow.BATCH_SIZE, "100");
+runner.enqueue(buildSeparatedString(ids, separatorProp), attrs);
+runner.run(1, true);
+
+Assert.assertTrue("The mock client was not empty.", 
hBaseClient.isEmpty());
+}
+
+@Test
+public void testDeletesSeparatedByNewLines() {
+testSeparatedDeletes("\n");
+}
+
+@Test
+public void testDeletesSeparatedByCommas() {
+testSeparatedDeletes(",");
+}
+
+@Test
+public void testDeleteWithELSeparator() {
+runner.setValidateExpressionUsage(true);
+Map attrs = new HashMap<>();
+attrs.put("test.separator", "");
+testSeparatedDeletes("${test.separator}", "", attrs);
+}
+
+@Test
+public void testDeleteWithExpressionLanguage() {
+List ids = 

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167854611
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
--- End diff --

Same comment as "rowkey.start" on the description applied here.


---

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167851239
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
+}
+)
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
--- End diff --

`@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)` should be 
added.


---

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167858912
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
+}
+)
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
+static final AllowableValue ROW_ID_BODY = new AllowableValue("body", 
"FlowFile content", "Get the row key(s) from the flowfile content.");
--- End diff --

I'd suggest keep the naming consistent. Both of `body` and `content` are 
used, but it would be cleaner if we use just one of them. If so, I'd prefer 
`content`.


---

[GitHub] nifi issue #2449: NIFI-4840 Fix formatting for expression language guide for...

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2449
  
+1, merging to master, thanks @Leah-Anderson @zenfenan 


---

[GitHub] nifi pull request #2449: NIFI-4840 Fix formatting for expression language gu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2449


---

[jira] [Commented] (NIFI-4840) Fix Formatting For Expression Language Guide (toRadix/fromRadix)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361984#comment-16361984
 ] 

ASF GitHub Bot commented on NIFI-4840:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2449


> Fix Formatting For Expression Language Guide (toRadix/fromRadix)
> 
>
> Key: NIFI-4840
> URL: https://issues.apache.org/jira/browse/NIFI-4840
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation  Website
>Affects Versions: 1.5.0
>Reporter: Leah Anderson
>Priority: Trivial
>  Labels: documentation
> Fix For: 1.6.0
>
> Attachments: Screenshot from 2018-02-04 19-26-40.png
>
>
> The documentation in the expression language guide is not formatted correctly 
> for the toRadix/fromRadix sections due to a misplaced # sign. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4840) Fix Formatting For Expression Language Guide (toRadix/fromRadix)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361982#comment-16361982
 ] 

ASF GitHub Bot commented on NIFI-4840:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2449
  
+1, merging to master, thanks @Leah-Anderson @zenfenan 


> Fix Formatting For Expression Language Guide (toRadix/fromRadix)
> 
>
> Key: NIFI-4840
> URL: https://issues.apache.org/jira/browse/NIFI-4840
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation  Website
>Affects Versions: 1.5.0
>Reporter: Leah Anderson
>Priority: Trivial
>  Labels: documentation
> Fix For: 1.6.0
>
> Attachments: Screenshot from 2018-02-04 19-26-40.png
>
>
> The documentation in the expression language guide is not formatted correctly 
> for the toRadix/fromRadix sections due to a misplaced # sign. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (NIFI-4840) Fix Formatting For Expression Language Guide (toRadix/fromRadix)

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-4840.
--
Resolution: Fixed

> Fix Formatting For Expression Language Guide (toRadix/fromRadix)
> 
>
> Key: NIFI-4840
> URL: https://issues.apache.org/jira/browse/NIFI-4840
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation  Website
>Affects Versions: 1.5.0
>Reporter: Leah Anderson
>Priority: Trivial
>  Labels: documentation
> Fix For: 1.6.0
>
> Attachments: Screenshot from 2018-02-04 19-26-40.png
>
>
> The documentation in the expression language guide is not formatted correctly 
> for the toRadix/fromRadix sections due to a misplaced # sign. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4840) Fix Formatting For Expression Language Guide (toRadix/fromRadix)

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361983#comment-16361983
 ] 

ASF subversion and git services commented on NIFI-4840:
---

Commit a3edd34db031d5d5efc83a90e99368a1dfd20a4f in nifi's branch 
refs/heads/master from [~Leah Anderson]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=a3edd34 ]

NIFI-4840 Fix formatting for expression language guide for toRadix and fromRadix

Signed-off-by: Pierre Villard 

This closes #2449.


> Fix Formatting For Expression Language Guide (toRadix/fromRadix)
> 
>
> Key: NIFI-4840
> URL: https://issues.apache.org/jira/browse/NIFI-4840
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation  Website
>Affects Versions: 1.5.0
>Reporter: Leah Anderson
>Priority: Trivial
>  Labels: documentation
> Fix For: 1.6.0
>
> Attachments: Screenshot from 2018-02-04 19-26-40.png
>
>
> The documentation in the expression language guide is not formatted correctly 
> for the toRadix/fromRadix sections due to a misplaced # sign. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4872) NIFI component high resource usage annotation

[Pierre Villard (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361970#comment-16361970
 ] 

Pierre Villard commented on NIFI-4872:
--

Would it also make sense to add a description field in this annotation? I'm 
thinking about Merge and Split processors: we often recall users to perform a 
two steps processing when using such a processor with a huge file. We could 
also update the capability description though.

> NIFI component high resource usage annotation
> -
>
> Key: NIFI-4872
> URL: https://issues.apache.org/jira/browse/NIFI-4872
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework, Core UI
>Affects Versions: 1.5.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Critical
>
> NiFi Processors currently have no means to relay whether or not they have may 
> be resource intensive or not. The idea here would be to introduce an 
> Annotation that can be added to Processors that indicate they may cause high 
> memory, disk, CPU, or network usage. For instance, any Processor that reads 
> the FlowFile contents into memory (like many XML Processors for instance) may 
> cause high memory usage. What ultimately determines if there is high 
> memory/disk/cpu/network usage will depend on the FlowFiles being processed. 
> With many of these components in the dataflow, it increases the risk of 
> OutOfMemoryErrors and performance degradation.
> The annotation should support one or more values from a fixed list of: CPU, 
> Disk, Memory, Network.
> By marking components with this new Annotation, we can update the generated 
> Processor documentation to include this fact.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4875) Import flow from nifi-registry on bootstrap/initial start

[Pierre Villard (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362040#comment-16362040
 ] 

Pierre Villard commented on NIFI-4875:
--

[~spotty] - are you requesting this feature for NiFi Docker image only or also 
the NiFi standard build ?

I see how this could be interesting for the Docker image and that should not be 
really hard to do: passing a variable environment to specify the URL of the 
registry, and if the variable exists, then we initialize the flow.xml.gz with a 
registry client that will reference the variable.

> Import flow from nifi-registry on bootstrap/initial start
> -
>
> Key: NIFI-4875
> URL: https://issues.apache.org/jira/browse/NIFI-4875
> Project: Apache NiFi
>  Issue Type: Wish
>  Components: Flow Versioning
>Affects Versions: 1.5.0
>Reporter: Daniel Oakley
>Priority: Major
>  Labels: registry
>
> As I user I would like to be able to automate the deployment of a new Nifi 
> instance (e.g. using Docker, Ansible), and have that instance load a specific 
> versioned flow from the nifi-registry and start running it automatically.
> To enable this I would like to be able to specify the nifi-registry URL in 
> nifi properties file, and the flow to load/start via either the properties 
> file or a command line option.
> This will greatly increase the ease of deploying nifi in automated 
> environments such as Kubernetes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4864) Additional Resources property pointing at a directory won't find new JARs

[zenfenan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zenfenan updated NIFI-4864:
---
Description: 
If you have a processor/Controller Service/Reporting Task that has a property 
with dynamicallyModifiesClasspath(true) and you set the value to a directory, 
the resources in that directory will only be calculated when that property 
changes. This means if you added JARs to the directory later, and stopped and 
started your processor, those new JARs still won't be available. You would have 
to change the property to a new directory, or back and forth to some other 
directory, to force a recalculation.

The setProperties method in AbstractConfiguredComponent is where it looks at 
incoming property changes and determines if any were for classpath related 
properties and then calls reload accordingly.

We would need to consider the case where setProperties is never even being 
called, someone just stops and starts the processor and would want to pick up 
any new JARs added.

A possible solution might be to computer some kind of hash/fingerprint of the 
URLs each time reload is called, and then when starting the processor we could 
recompute the fingerprint and compare it to the previous one. If they are 
different then we call reload before starting the component.

  was:
If you have a processor that has a property with 
dynamicallyModifiesClasspath(true) and you set the value to a directory, the 
resources in that directory will only be calculated when that property changes. 
This means if you added JARs to the directory later, and stopped and started 
your processor, those new JARs still won't be available. You would have to 
change the property to a new directory, or back and forth to some other 
directory, to force a recalculation.

The setProperties method in AbstractConfiguredComponent is where it looks at 
incoming property changes and determines if any were for classpath related 
properties and then calls reload accordingly.

We would need to consider the case where setProperties is never even being 
called, someone just stops and starts the processor and would want to pick up 
any new JARs added.

A possible solution might be to computer some kind of hash/fingerprint of the 
URLs each time reload is called, and then when starting the processor we could 
recompute the fingerprint and compare it to the previous one. If they are 
different then we call reload before starting the component.


> Additional Resources property pointing at a directory won't find new JARs
> -
>
> Key: NIFI-4864
> URL: https://issues.apache.org/jira/browse/NIFI-4864
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0
>Reporter: Bryan Bende
>Assignee: zenfenan
>Priority: Minor
>
> If you have a processor/Controller Service/Reporting Task that has a property 
> with dynamicallyModifiesClasspath(true) and you set the value to a directory, 
> the resources in that directory will only be calculated when that property 
> changes. This means if you added JARs to the directory later, and stopped and 
> started your processor, those new JARs still won't be available. You would 
> have to change the property to a new directory, or back and forth to some 
> other directory, to force a recalculation.
> The setProperties method in AbstractConfiguredComponent is where it looks at 
> incoming property changes and determines if any were for classpath related 
> properties and then calls reload accordingly.
> We would need to consider the case where setProperties is never even being 
> called, someone just stops and starts the processor and would want to pick up 
> any new JARs added.
> A possible solution might be to computer some kind of hash/fingerprint of the 
> URLs each time reload is called, and then when starting the processor we 
> could recompute the fingerprint and compare it to the previous one. If they 
> are different then we call reload before starting the component.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (MINIFICPP-400) Identify workaround for CURL ABI incompatibilities.

[marco polo (JIRA)]

marco polo created MINIFICPP-400:


 Summary: Identify workaround for CURL ABI incompatibilities. 
 Key: MINIFICPP-400
 URL: https://issues.apache.org/jira/browse/MINIFICPP-400
 Project: NiFi MiNiFi C++
  Issue Type: Improvement
Reporter: marco polo
Assignee: marco polo






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4819) Add support to delete blob from Azure Storage container

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362415#comment-16362415
 ] 

ASF GitHub Bot commented on NIFI-4819:
--

Github user zenfenan commented on the issue:

https://github.com/apache/nifi/pull/2436
  
@pvillard31 Actually I left `AtomicReference` on purpose. The reason was I 
wanted to implement some features later that would need that but yeah it 
doesn't concern or matter for the current commit/Jira so I took that off as 
well as fixed to `@InputRequirement(Requirement.INPUT_REQUIRED)`


> Add support to delete blob from Azure Storage container
> ---
>
> Key: NIFI-4819
> URL: https://issues.apache.org/jira/browse/NIFI-4819
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: zenfenan
>Priority: Major
>
> Implement a delete processor that handles deleting blob from Azure Storage 
> container. This should be an extension of nifi-azure-nar bundle. Currently, 
> the azure bundle's storage processors has support to list, fetch, put Azure 
> Storage blobs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2436: NIFI-4819 Added support to delete blob from Azure Storage ...

[zenfenan]

Github user zenfenan commented on the issue:

https://github.com/apache/nifi/pull/2436
  
@pvillard31 Actually I left `AtomicReference` on purpose. The reason was I 
wanted to implement some features later that would need that but yeah it 
doesn't concern or matter for the current commit/Jira so I took that off as 
well as fixed to `@InputRequirement(Requirement.INPUT_REQUIRED)`


---

[jira] [Updated] (NIFI-4869) Update SplitXml data handling

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-4869:

Status: Patch Available  (was: In Progress)

> Update SplitXml data handling
> -
>
> Key: NIFI-4869
> URL: https://issues.apache.org/jira/browse/NIFI-4869
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: processor, xml
>
> Change the way {{SplitXml}} handles incoming data. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167939817
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
+}
+)
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
--- End diff --

Done


---

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362706#comment-16362706
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167940422
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
--- End diff --

Done.


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362707#comment-16362707
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167940501
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
--- End diff --

Done


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167943515
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
+}
+)
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
+static final AllowableValue ROW_ID_BODY = new AllowableValue("body", 
"FlowFile content", "Get the row key(s) from the flowfile content.");
+static final AllowableValue ROW_ID_ATTR = new AllowableValue("attr", 
"FlowFile attributes", "Get the row key from an expression language 
statement.");
+
+static final String RESTART_INDEX = "restart.index";
+static final String ROWKEY_START = "rowkey.start";
+static final String ROWKEY_END   = "rowkey.end";
+
+static final PropertyDescriptor ROW_ID_LOCATION = new 
PropertyDescriptor.Builder()
+.name("delete-hb-row-id-location")
+.displayName("Row ID Location")
+.description("The location of the row ID to use for building 
the delete. Can be from the content or an expression language statement.")
+.required(true)
+.defaultValue(ROW_ID_BODY.getValue())
+.allowableValues(ROW_ID_BODY, ROW_ID_ATTR)
+.addValidator(Validator.VALID)
+.build();
+
+static final PropertyDescriptor FLOWFILE_FETCH_COUNT = new 
PropertyDescriptor.Builder()
+.name("delete-hb-flowfile-fetch-count")
+.displayName("Flowfile Fetch Count")
+.description("The number of flowfiles to fetch per run.")
+.required(true)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.defaultValue("5")
+.expressionLanguageSupported(false)
+.build();
+
+static final PropertyDescriptor BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("delete-hb-batch-size")
+.displayName("Batch Size")
+.description("The number of deletes to send per batch.")
+.required(true)
+.defaultValue("50")
+

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362724#comment-16362724
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167943515
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
+}
+)
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
+static final AllowableValue ROW_ID_BODY = new AllowableValue("body", 
"FlowFile content", "Get the row key(s) from the flowfile content.");
+static final AllowableValue ROW_ID_ATTR = new AllowableValue("attr", 
"FlowFile attributes", "Get the row key from an expression language 
statement.");
+
+static final String RESTART_INDEX = "restart.index";
+static final String ROWKEY_START = "rowkey.start";
+static final String ROWKEY_END   = "rowkey.end";
+
+static final PropertyDescriptor ROW_ID_LOCATION = new 
PropertyDescriptor.Builder()
+.name("delete-hb-row-id-location")
+.displayName("Row ID Location")
+.description("The location of the row ID to use for building 
the delete. Can be from the content or an expression language statement.")
+.required(true)
+.defaultValue(ROW_ID_BODY.getValue())
+.allowableValues(ROW_ID_BODY, ROW_ID_ATTR)
+.addValidator(Validator.VALID)
+.build();
+
+static final PropertyDescriptor FLOWFILE_FETCH_COUNT = new 
PropertyDescriptor.Builder()
+.name("delete-hb-flowfile-fetch-count")
+.displayName("Flowfile Fetch Count")
+.description("The number of flowfiles to fetch per run.")
+.required(true)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.defaultValue("5")
+.expressionLanguageSupported(false)
+.build();
+
+static final PropertyDescriptor BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("delete-hb-batch-size")
+   

[jira] [Commented] (NIFI-4862) Add Copy original attributtes to SelectHiveQL processer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362824#comment-16362824
 ] 

ASF GitHub Bot commented on NIFI-4862:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2467#discussion_r167955630
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -261,6 +273,7 @@ public void onTrigger(ProcessContext context, 
ProcessSessionFactory sessionFacto
 
 private void onTrigger(final ProcessContext context, final 
ProcessSession session) throws ProcessException {
 FlowFile fileToProcess = (context.hasIncomingConnection() ? 
session.get() : null);
+Attributes originalAttributes;
--- End diff --

This variable is unused. 


> Add Copy original attributtes to SelectHiveQL processer
> ---
>
> Key: NIFI-4862
> URL: https://issues.apache.org/jira/browse/NIFI-4862
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Jakub Leś
>Priority: Minor
> Attachments: 
> 0001-NIFI-4862-Add-Copy-original-attributtes-to-SelectHiv.patch
>
>
> Hi, 
> Please add "Copy original attributes" to processor SelectHiveQL. Thanks to 
> that we can use HttpRequest and HttpResponse to synchronize fetching query 
> result in webservice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4862) Add Copy original attributtes to SelectHiveQL processer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362825#comment-16362825
 ] 

ASF GitHub Bot commented on NIFI-4862:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2467#discussion_r167955721
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -306,6 +319,7 @@ private void onTrigger(final ProcessContext context, 
final ProcessSession sessio
 final boolean quote = 
context.getProperty(HIVEQL_CSV_QUOTE).asBoolean();
 final boolean escape = 
context.getProperty(HIVEQL_CSV_HEADER).asBoolean();
 final String fragmentIdentifier = UUID.randomUUID().toString();
+final boolean copyAttributtes = 
context.getProperty(HIVEQL_COPY_ATRIBUTTES).asBoolean();
--- End diff --

Please correct typo in the variable name -- `copyAttributes`. 


> Add Copy original attributtes to SelectHiveQL processer
> ---
>
> Key: NIFI-4862
> URL: https://issues.apache.org/jira/browse/NIFI-4862
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Jakub Leś
>Priority: Minor
> Attachments: 
> 0001-NIFI-4862-Add-Copy-original-attributtes-to-SelectHiv.patch
>
>
> Hi, 
> Please add "Copy original attributes" to processor SelectHiveQL. Thanks to 
> that we can use HttpRequest and HttpResponse to synchronize fetching query 
> result in webservice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2467: NIFI-4862 Add Copy original attributtes to SelectHi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2467#discussion_r167957594
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -437,6 +451,16 @@ private void onTrigger(final ProcessContext context, 
final ProcessSession sessio
 resultSetFlowFiles.set(i,
 
session.putAttribute(resultSetFlowFiles.get(i), "fragment.count", 
Integer.toString(fragmentIndex)));
 }
+
+if(copyAttributtes)
+{
+for(Map.Entry entry : 
fileToProcess.getAttributes().entrySet()) {
--- End diff --

The attributes on the original flowfile are static, but this logic is 
placed inside a loop which iterates over every resulting flowfile (there may be 
many) and performs the same extraction multiple times. Please move the 
retrieval of the original attributes outside of the loop and store them in a 
local variable, then use `session.putAllAttributes(resultSetFlowFiles.get(i), 
map);` inside the loop to copy all the original attributes into the new 
flowfile. 


---

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167940095
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/test/java/org/apache/nifi/hbase/TestDeleteHBaseRow.java
 ---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+public class TestDeleteHBaseRow {
+private TestRunner runner;
+private MockHBaseClientService hBaseClient;
+
+@Before
+public void setup() throws InitializationException {
+runner = TestRunners.newTestRunner(new DeleteHBaseRow());
+
+hBaseClient = new MockHBaseClientService();
+runner.addControllerService("hbaseClient", hBaseClient);
+runner.enableControllerService(hBaseClient);
+
+runner.setProperty(DeleteHBaseRow.TABLE_NAME, "nifi");
+runner.setProperty(DeleteHBaseRow.HBASE_CLIENT_SERVICE, 
"hbaseClient");
+}
+
+List populateTable(int max) {
+List ids = new ArrayList<>();
+for (int index = 0; index < max; index++) {
+String uuid = UUID.randomUUID().toString();
+ids.add(uuid);
+Map cells = new HashMap<>();
+cells.put("test", UUID.randomUUID().toString());
+hBaseClient.addResult(uuid, cells, System.currentTimeMillis());
+}
+
+return ids;
+}
+
+@Test
+public void testSimpleDelete() {
+List ids = populateTable(100);
+
+runner.setProperty(DeleteHBaseRow.BATCH_SIZE, "100");
+runner.setProperty(DeleteHBaseRow.FLOWFILE_FETCH_COUNT, "100");
+for (String id : ids) {
+runner.enqueue(id);
+}
+
+runner.run(1, true);
+Assert.assertTrue("The mock client was not empty.", 
hBaseClient.isEmpty());
+}
+
+private String buildSeparatedString(List ids, String 
separator) {
+StringBuilder sb = new StringBuilder();
+for (int index = 1; index <= ids.size(); index++) {
+sb.append(ids.get(index - 1)).append(separator);
+}
+
+return sb.toString();
+}
+
+private void testSeparatedDeletes(String separator) {
+testSeparatedDeletes(separator, separator, new HashMap());
+}
+
+private void testSeparatedDeletes(String separator, String 
separatorProp, Map attrs) {
+List ids = populateTable(1);
+runner.setProperty(DeleteHBaseRow.KEY_SEPARATOR, separator);
+runner.setProperty(DeleteHBaseRow.BATCH_SIZE, "100");
+runner.enqueue(buildSeparatedString(ids, separatorProp), attrs);
+runner.run(1, true);
+
+Assert.assertTrue("The mock client was not empty.", 
hBaseClient.isEmpty());
+}
+
+@Test
+public void testDeletesSeparatedByNewLines() {
+testSeparatedDeletes("\n");
+}
+
+@Test
+public void testDeletesSeparatedByCommas() {
+testSeparatedDeletes(",");
+}
+
+@Test
+public void testDeleteWithELSeparator() {
+runner.setValidateExpressionUsage(true);
+Map attrs = new HashMap<>();
+attrs.put("test.separator", "");
+testSeparatedDeletes("${test.separator}", "", attrs);
+}
+
+@Test
+public void testDeleteWithExpressionLanguage() {
+List ids = populateTable(1000);

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362700#comment-16362700
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167939817
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
+}
+)
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
--- End diff --

Done


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2294: NIFI-3538 Added DeleteHBaseRow

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2294
  
@ijokarumawak All of your feedback should now be addressed.


---

[jira] [Commented] (NIFI-4862) Add Copy original attributtes to SelectHiveQL processer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362831#comment-16362831
 ] 

ASF GitHub Bot commented on NIFI-4862:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2467#discussion_r167956282
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -437,6 +451,16 @@ private void onTrigger(final ProcessContext context, 
final ProcessSession sessio
 resultSetFlowFiles.set(i,
 
session.putAttribute(resultSetFlowFiles.get(i), "fragment.count", 
Integer.toString(fragmentIndex)));
 }
+
+if(copyAttributtes)
--- End diff --

Please follow the [style guide located 
here](https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-CodeStyle)
 for consistency. 


> Add Copy original attributtes to SelectHiveQL processer
> ---
>
> Key: NIFI-4862
> URL: https://issues.apache.org/jira/browse/NIFI-4862
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Jakub Leś
>Priority: Minor
> Attachments: 
> 0001-NIFI-4862-Add-Copy-original-attributtes-to-SelectHiv.patch
>
>
> Hi, 
> Please add "Copy original attributes" to processor SelectHiveQL. Thanks to 
> that we can use HttpRequest and HttpResponse to synchronize fetching query 
> result in webservice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362704#comment-16362704
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167940095
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/test/java/org/apache/nifi/hbase/TestDeleteHBaseRow.java
 ---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+public class TestDeleteHBaseRow {
+private TestRunner runner;
+private MockHBaseClientService hBaseClient;
+
+@Before
+public void setup() throws InitializationException {
+runner = TestRunners.newTestRunner(new DeleteHBaseRow());
+
+hBaseClient = new MockHBaseClientService();
+runner.addControllerService("hbaseClient", hBaseClient);
+runner.enableControllerService(hBaseClient);
+
+runner.setProperty(DeleteHBaseRow.TABLE_NAME, "nifi");
+runner.setProperty(DeleteHBaseRow.HBASE_CLIENT_SERVICE, 
"hbaseClient");
+}
+
+List populateTable(int max) {
+List ids = new ArrayList<>();
+for (int index = 0; index < max; index++) {
+String uuid = UUID.randomUUID().toString();
+ids.add(uuid);
+Map cells = new HashMap<>();
+cells.put("test", UUID.randomUUID().toString());
+hBaseClient.addResult(uuid, cells, System.currentTimeMillis());
+}
+
+return ids;
+}
+
+@Test
+public void testSimpleDelete() {
+List ids = populateTable(100);
+
+runner.setProperty(DeleteHBaseRow.BATCH_SIZE, "100");
+runner.setProperty(DeleteHBaseRow.FLOWFILE_FETCH_COUNT, "100");
+for (String id : ids) {
+runner.enqueue(id);
+}
+
+runner.run(1, true);
+Assert.assertTrue("The mock client was not empty.", 
hBaseClient.isEmpty());
+}
+
+private String buildSeparatedString(List ids, String 
separator) {
+StringBuilder sb = new StringBuilder();
+for (int index = 1; index <= ids.size(); index++) {
+sb.append(ids.get(index - 1)).append(separator);
+}
+
+return sb.toString();
+}
+
+private void testSeparatedDeletes(String separator) {
+testSeparatedDeletes(separator, separator, new HashMap());
+}
+
+private void testSeparatedDeletes(String separator, String 
separatorProp, Map attrs) {
+List ids = populateTable(1);
+runner.setProperty(DeleteHBaseRow.KEY_SEPARATOR, separator);
+runner.setProperty(DeleteHBaseRow.BATCH_SIZE, "100");
+runner.enqueue(buildSeparatedString(ids, separatorProp), attrs);
+runner.run(1, true);
+
+Assert.assertTrue("The mock client was not empty.", 
hBaseClient.isEmpty());
+}
+
+@Test
+public void testDeletesSeparatedByNewLines() {
+testSeparatedDeletes("\n");
+}
+
+@Test
+public void testDeletesSeparatedByCommas() {
+testSeparatedDeletes(",");
+}
+
+@Test
+public void testDeleteWithELSeparator() {
+runner.setValidateExpressionUsage(true);
+Map attrs = new HashMap<>();
+

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167940422
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
--- End diff --

Done.


---

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167940501
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
--- End diff --

Done


---

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362734#comment-16362734
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2294
  
@ijokarumawak All of your feedback should now be addressed.


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2467: NIFI-4862 Add Copy original attributtes to SelectHi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2467#discussion_r167955721
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -306,6 +319,7 @@ private void onTrigger(final ProcessContext context, 
final ProcessSession sessio
 final boolean quote = 
context.getProperty(HIVEQL_CSV_QUOTE).asBoolean();
 final boolean escape = 
context.getProperty(HIVEQL_CSV_HEADER).asBoolean();
 final String fragmentIdentifier = UUID.randomUUID().toString();
+final boolean copyAttributtes = 
context.getProperty(HIVEQL_COPY_ATRIBUTTES).asBoolean();
--- End diff --

Please correct typo in the variable name -- `copyAttributes`. 


---

[jira] [Commented] (NIFI-4862) Add Copy original attributtes to SelectHiveQL processer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362822#comment-16362822
 ] 

ASF GitHub Bot commented on NIFI-4862:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2467#discussion_r167955177
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -32,7 +32,7 @@
 import java.util.UUID;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicLong;
-
+import java.util.jar.Attributes;
--- End diff --

I think this import is unused. 


> Add Copy original attributtes to SelectHiveQL processer
> ---
>
> Key: NIFI-4862
> URL: https://issues.apache.org/jira/browse/NIFI-4862
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Jakub Leś
>Priority: Minor
> Attachments: 
> 0001-NIFI-4862-Add-Copy-original-attributtes-to-SelectHiv.patch
>
>
> Hi, 
> Please add "Copy original attributes" to processor SelectHiveQL. Thanks to 
> that we can use HttpRequest and HttpResponse to synchronize fetching query 
> result in webservice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2467: NIFI-4862 Add Copy original attributtes to SelectHi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2467#discussion_r167955428
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -203,6 +203,16 @@
 .expressionLanguageSupported(false)
 .build();
 
+public static final PropertyDescriptor HIVEQL_COPY_ATRIBUTTES = new 
PropertyDescriptor.Builder()
--- End diff --

Please correct typos in the field name and in `.displayName()`, 
`.description()`, and `.name()` -- "attributes". 


---

[jira] [Commented] (NIFI-4862) Add Copy original attributtes to SelectHiveQL processer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362823#comment-16362823
 ] 

ASF GitHub Bot commented on NIFI-4862:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2467#discussion_r167955428
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -203,6 +203,16 @@
 .expressionLanguageSupported(false)
 .build();
 
+public static final PropertyDescriptor HIVEQL_COPY_ATRIBUTTES = new 
PropertyDescriptor.Builder()
--- End diff --

Please correct typos in the field name and in `.displayName()`, 
`.description()`, and `.name()` -- "attributes". 


> Add Copy original attributtes to SelectHiveQL processer
> ---
>
> Key: NIFI-4862
> URL: https://issues.apache.org/jira/browse/NIFI-4862
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Jakub Leś
>Priority: Minor
> Attachments: 
> 0001-NIFI-4862-Add-Copy-original-attributtes-to-SelectHiv.patch
>
>
> Hi, 
> Please add "Copy original attributes" to processor SelectHiveQL. Thanks to 
> that we can use HttpRequest and HttpResponse to synchronize fetching query 
> result in webservice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2467: NIFI-4862 Add Copy original attributtes to SelectHi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2467#discussion_r167955630
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -261,6 +273,7 @@ public void onTrigger(ProcessContext context, 
ProcessSessionFactory sessionFacto
 
 private void onTrigger(final ProcessContext context, final 
ProcessSession session) throws ProcessException {
 FlowFile fileToProcess = (context.hasIncomingConnection() ? 
session.get() : null);
+Attributes originalAttributes;
--- End diff --

This variable is unused. 


---

[jira] [Commented] (NIFI-4862) Add Copy original attributtes to SelectHiveQL processer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362849#comment-16362849
 ] 

ASF GitHub Bot commented on NIFI-4862:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2467
  
Thanks for your first contribution to Apache NiFi. I have made some inline 
comments on the code. Please also add a unit test covering the new 
functionality you added. You can see existing unit tests for this code in 
[`TestSelectHiveQL`](https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/test/java/org/apache/nifi/processors/hive/TestSelectHiveQL.java#L63).
 Our [Contributor 
Guide](https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide) also 
has helpful information to make your submission review go as smoothly as 
possible. 


> Add Copy original attributtes to SelectHiveQL processer
> ---
>
> Key: NIFI-4862
> URL: https://issues.apache.org/jira/browse/NIFI-4862
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Jakub Leś
>Priority: Minor
> Attachments: 
> 0001-NIFI-4862-Add-Copy-original-attributtes-to-SelectHiv.patch
>
>
> Hi, 
> Please add "Copy original attributes" to processor SelectHiveQL. Thanks to 
> that we can use HttpRequest and HttpResponse to synchronize fetching query 
> result in webservice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2467: NIFI-4862 Add Copy original attributtes to SelectHiveQL pr...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2467
  
Thanks for your first contribution to Apache NiFi. I have made some inline 
comments on the code. Please also add a unit test covering the new 
functionality you added. You can see existing unit tests for this code in 
[`TestSelectHiveQL`](https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/test/java/org/apache/nifi/processors/hive/TestSelectHiveQL.java#L63).
 Our [Contributor 
Guide](https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide) also 
has helpful information to make your submission review go as smoothly as 
possible. 


---

[GitHub] nifi pull request #2467: NIFI-4862 Add Copy original attributtes to SelectHi...

[jles01]

GitHub user jles01 opened a pull request:

https://github.com/apache/nifi/pull/2467

NIFI-4862 Add Copy original attributtes to SelectHiveQL processer

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jles01/nifi-1 nifi-4862

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2467.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2467


commit efa37937980a300f6362118fcea50c8469c31782
Author: Jakub Leś 
Date:   2018-02-13T16:14:14Z

NIFI-4862 Add Copy original attributtes to SelectHiveQL processer




---

[jira] [Commented] (NIFI-4862) Add Copy original attributtes to SelectHiveQL processer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362720#comment-16362720
 ] 

ASF GitHub Bot commented on NIFI-4862:
--

GitHub user jles01 opened a pull request:

https://github.com/apache/nifi/pull/2467

NIFI-4862 Add Copy original attributtes to SelectHiveQL processer

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jles01/nifi-1 nifi-4862

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2467.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2467


commit efa37937980a300f6362118fcea50c8469c31782
Author: Jakub Leś 
Date:   2018-02-13T16:14:14Z

NIFI-4862 Add Copy original attributtes to SelectHiveQL processer




> Add Copy original attributtes to SelectHiveQL processer
> ---
>
> Key: NIFI-4862
> URL: https://issues.apache.org/jira/browse/NIFI-4862
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Jakub Leś
>Priority: Minor
> Attachments: 
> 0001-NIFI-4862-Add-Copy-original-attributtes-to-SelectHiv.patch
>
>
> Hi, 
> Please add "Copy original attributes" to processor SelectHiveQL. Thanks to 
> that we can use HttpRequest and HttpResponse to synchronize fetching query 
> result in webservice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2468: NIFI-4849: Implemented REST Endpoint and associated...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2468

NIFI-4849: Implemented REST Endpoint and associated backend code to g…

…enerate a Diagnostics Report for a Processor

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4849

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2468.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2468


commit 9258cdd81c854d0e1a130a67f02913dc2c56dd16
Author: Mark Payne 
Date:   2018-02-02T17:16:36Z

NIFI-4849: Implemented REST Endpoint and associated backend code to 
generate a Diagnostics Report for a Processor




---

[jira] [Commented] (NIFI-4849) Add REST Endpoint for gathering Processor Diagnostics information

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362728#comment-16362728
 ] 

ASF GitHub Bot commented on NIFI-4849:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2468

NIFI-4849: Implemented REST Endpoint and associated backend code to g…

…enerate a Diagnostics Report for a Processor

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4849

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2468.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2468


commit 9258cdd81c854d0e1a130a67f02913dc2c56dd16
Author: Mark Payne 
Date:   2018-02-02T17:16:36Z

NIFI-4849: Implemented REST Endpoint and associated backend code to 
generate a Diagnostics Report for a Processor




> Add REST Endpoint for gathering Processor Diagnostics information
> -
>
> Key: NIFI-4849
> URL: https://issues.apache.org/jira/browse/NIFI-4849
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.6.0
>
>
> We need to add a REST endpoint that will use the appropriate resources to 
> gather the Processor Diagnostics information. Information to return should 
> include things like:
>  * Processor config
>  * Processor status
>  * Garbage Collection info
>  * Repo Sizes
>  * Connection info for connections whose source or destination is the 
> processor
>  * Controller Services that the processor is referencing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4849) Add REST Endpoint for gathering Processor Diagnostics information

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4849:
-
Fix Version/s: 1.6.0
   Status: Patch Available  (was: Open)

> Add REST Endpoint for gathering Processor Diagnostics information
> -
>
> Key: NIFI-4849
> URL: https://issues.apache.org/jira/browse/NIFI-4849
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.6.0
>
>
> We need to add a REST endpoint that will use the appropriate resources to 
> gather the Processor Diagnostics information. Information to return should 
> include things like:
>  * Processor config
>  * Processor status
>  * Garbage Collection info
>  * Repo Sizes
>  * Connection info for connections whose source or destination is the 
> processor
>  * Controller Services that the processor is referencing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362716#comment-16362716
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167941736
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
+}
+)
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
+static final AllowableValue ROW_ID_BODY = new AllowableValue("body", 
"FlowFile content", "Get the row key(s) from the flowfile content.");
--- End diff --

Done


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167941736
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttributes(
+value = {
+@WritesAttribute( attribute = "restart.index", description = "If a 
delete batch fails, 'restart.index' attribute is added to the FlowFile and sent 
to 'failure' " +
+"relationship, so that this processor can retry from there 
when the same FlowFile is routed again." ),
+@WritesAttribute( attribute = "rowkey.start", description = "The 
first rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes."),
+@WritesAttribute( attribute = "rowkey.end", description = "The 
last rowkey in the flowfile. Only written when a flowfile contains multiple 
deletes.")
+}
+)
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
+static final AllowableValue ROW_ID_BODY = new AllowableValue("body", 
"FlowFile content", "Get the row key(s) from the flowfile content.");
--- End diff --

Done


---

[GitHub] nifi pull request #2467: NIFI-4862 Add Copy original attributtes to SelectHi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2467#discussion_r167955177
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -32,7 +32,7 @@
 import java.util.UUID;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicLong;
-
+import java.util.jar.Attributes;
--- End diff --

I think this import is unused. 


---

[GitHub] nifi pull request #2467: NIFI-4862 Add Copy original attributtes to SelectHi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2467#discussion_r167956282
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -437,6 +451,16 @@ private void onTrigger(final ProcessContext context, 
final ProcessSession sessio
 resultSetFlowFiles.set(i,
 
session.putAttribute(resultSetFlowFiles.get(i), "fragment.count", 
Integer.toString(fragmentIndex)));
 }
+
+if(copyAttributtes)
--- End diff --

Please follow the [style guide located 
here](https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-CodeStyle)
 for consistency. 


---

[jira] [Commented] (NIFI-4862) Add Copy original attributtes to SelectHiveQL processer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362842#comment-16362842
 ] 

ASF GitHub Bot commented on NIFI-4862:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2467#discussion_r167957594
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -437,6 +451,16 @@ private void onTrigger(final ProcessContext context, 
final ProcessSession sessio
 resultSetFlowFiles.set(i,
 
session.putAttribute(resultSetFlowFiles.get(i), "fragment.count", 
Integer.toString(fragmentIndex)));
 }
+
+if(copyAttributtes)
+{
+for(Map.Entry entry : 
fileToProcess.getAttributes().entrySet()) {
--- End diff --

The attributes on the original flowfile are static, but this logic is 
placed inside a loop which iterates over every resulting flowfile (there may be 
many) and performs the same extraction multiple times. Please move the 
retrieval of the original attributes outside of the loop and store them in a 
local variable, then use `session.putAllAttributes(resultSetFlowFiles.get(i), 
map);` inside the loop to copy all the original attributes into the new 
flowfile. 


> Add Copy original attributtes to SelectHiveQL processer
> ---
>
> Key: NIFI-4862
> URL: https://issues.apache.org/jira/browse/NIFI-4862
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Jakub Leś
>Priority: Minor
> Attachments: 
> 0001-NIFI-4862-Add-Copy-original-attributtes-to-SelectHiv.patch
>
>
> Hi, 
> Please add "Copy original attributes" to processor SelectHiveQL. Thanks to 
> that we can use HttpRequest and HttpResponse to synchronize fetching query 
> result in webservice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2469: NIFI-4870 Upgraded activemq-client and activemq-bro...

[alopresto]

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/2469

NIFI-4870 Upgraded activemq-client and activemq-broker versions to 5.…

…15.3.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-4870

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2469.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2469


commit 3aea6f99e5562188e5959d4461978f35c9ea8262
Author: Andy LoPresto 
Date:   2018-02-13T20:09:15Z

NIFI-4870 Upgraded activemq-client and activemq-broker versions to 5.15.3.




---

[GitHub] nifi issue #2466: NIFI-4869

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2466
  
Thanks @alopresto! This has been merged to master.


---

[jira] [Commented] (NIFI-4869) Update SplitXml data handling

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363000#comment-16363000
 ] 

ASF GitHub Bot commented on NIFI-4869:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2466


> Update SplitXml data handling
> -
>
> Key: NIFI-4869
> URL: https://issues.apache.org/jira/browse/NIFI-4869
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: processor, xml
>
> Change the way {{SplitXml}} handles incoming data. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4869) Update SplitXml data handling

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362999#comment-16362999
 ] 

ASF GitHub Bot commented on NIFI-4869:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2466
  
Thanks @alopresto! This has been merged to master.


> Update SplitXml data handling
> -
>
> Key: NIFI-4869
> URL: https://issues.apache.org/jira/browse/NIFI-4869
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: processor, xml
>
> Change the way {{SplitXml}} handles incoming data. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4869) Update SplitXml data handling

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362997#comment-16362997
 ] 

ASF subversion and git services commented on NIFI-4869:
---

Commit 28067a29fd13cdf8e21b440fc65c6dd67872522f in nifi's branch 
refs/heads/master from [~alopresto]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=28067a2 ]

NIFI-4869 Added SAX utility method for SplitXML processor.
Added unit tests.
Added test resources to RAT exclude list.
This closes #2466


> Update SplitXml data handling
> -
>
> Key: NIFI-4869
> URL: https://issues.apache.org/jira/browse/NIFI-4869
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: processor, xml
>
> Change the way {{SplitXml}} handles incoming data. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2466: NIFI-4869

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2466


---

[GitHub] nifi issue #2468: NIFI-4849: Implemented REST Endpoint and associated backen...

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2468
  
Will review...


---

[jira] [Commented] (NIFI-4849) Add REST Endpoint for gathering Processor Diagnostics information

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362885#comment-16362885
 ] 

ASF GitHub Bot commented on NIFI-4849:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2468
  
Will review...


> Add REST Endpoint for gathering Processor Diagnostics information
> -
>
> Key: NIFI-4849
> URL: https://issues.apache.org/jira/browse/NIFI-4849
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.6.0
>
>
> We need to add a REST endpoint that will use the appropriate resources to 
> gather the Processor Diagnostics information. Information to return should 
> include things like:
>  * Processor config
>  * Processor status
>  * Garbage Collection info
>  * Repo Sizes
>  * Connection info for connections whose source or destination is the 
> processor
>  * Controller Services that the processor is referencing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2427: MINIFI-4788 Exposing nifi.web.proxy.host configuration for...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2427
  
I think the Jira this should be linked to is 
[NIFI-4788](https://issues.apache.org/jira/browse/NIFI-4788). 


---

[GitHub] nifi issue #2427: MINIFI-4788 Exposing nifi.web.proxy.host configuration for...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2427
  
Reviewing...


---

[jira] [Commented] (NIFI-4870) Upgrade ActiveMQ dependencies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362983#comment-16362983
 ] 

ASF GitHub Bot commented on NIFI-4870:
--

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/2469

NIFI-4870 Upgraded activemq-client and activemq-broker versions to 5.…

…15.3.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-4870

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2469.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2469


commit 3aea6f99e5562188e5959d4461978f35c9ea8262
Author: Andy LoPresto 
Date:   2018-02-13T20:09:15Z

NIFI-4870 Upgraded activemq-client and activemq-broker versions to 5.15.3.




> Upgrade ActiveMQ dependencies
> -
>
> Key: NIFI-4870
> URL: https://issues.apache.org/jira/browse/NIFI-4870
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: activemq, dependencies, jms
>
> Upgrade the version of {{activemq-client}} to 5.14.x. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4869) Update SplitXml data handling

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-4869:
--
   Resolution: Fixed
Fix Version/s: 1.6.0
   Status: Resolved  (was: Patch Available)

> Update SplitXml data handling
> -
>
> Key: NIFI-4869
> URL: https://issues.apache.org/jira/browse/NIFI-4869
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: processor, xml
> Fix For: 1.6.0
>
>
> Change the way {{SplitXml}} handles incoming data. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4872) NIFI component high resource usage annotation

[Pierre Villard (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363069#comment-16363069
 ] 

Pierre Villard commented on NIFI-4872:
--

[~jtstorck] - cool! Sounds good to me!

> NIFI component high resource usage annotation
> -
>
> Key: NIFI-4872
> URL: https://issues.apache.org/jira/browse/NIFI-4872
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework, Core UI
>Affects Versions: 1.5.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Critical
>
> NiFi Processors currently have no means to relay whether or not they have may 
> be resource intensive or not. The idea here would be to introduce an 
> Annotation that can be added to Processors that indicate they may cause high 
> memory, disk, CPU, or network usage. For instance, any Processor that reads 
> the FlowFile contents into memory (like many XML Processors for instance) may 
> cause high memory usage. What ultimately determines if there is high 
> memory/disk/cpu/network usage will depend on the FlowFiles being processed. 
> With many of these components in the dataflow, it increases the risk of 
> OutOfMemoryErrors and performance degradation.
> The annotation should support one value from a fixed list of: CPU, Disk, 
> Memory, Network.  It should also allow the developer to provide a custom 
> description of the scenario that the component would fall under the high 
> usage category.  The annotation should be able to be specified multiple 
> times, for as many resources as it has the potential to be high usage.
> By marking components with this new Annotation, we can update the generated 
> Processor documentation to include this fact.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4875) Import flow from nifi-registry on bootstrap/initial start

[Pierre Villard (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363086#comment-16363086
 ] 

Pierre Villard commented on NIFI-4875:
--

What you describe is per-design: it's best to isolate independant workflows in 
separated process groups and it's not possible to save the root process group 
in the registry (don't have in mind the specifics around this decision but, in 
a multi tenant environment, you probably would not expect to version everything 
when multiple teams are working on multiple workflows).

Regarding your initial ask, using the REST API or the CLI under development 
(NIFI-4839), it should be easy to import a given version of a given workflow in 
a given bucket. It could be called when launching the Docker image.

> Import flow from nifi-registry on bootstrap/initial start
> -
>
> Key: NIFI-4875
> URL: https://issues.apache.org/jira/browse/NIFI-4875
> Project: Apache NiFi
>  Issue Type: Wish
>  Components: Flow Versioning
>Affects Versions: 1.5.0
>Reporter: Daniel Oakley
>Priority: Major
>  Labels: registry
>
> As I user I would like to be able to automate the deployment of a new Nifi 
> instance (e.g. using Docker, Ansible), and have that instance load a specific 
> versioned flow from the nifi-registry and start running it automatically.
> To enable this I would like to be able to specify the nifi-registry URL in 
> nifi properties file, and the flow to load/start via either the properties 
> file or a command line option.
> This will greatly increase the ease of deploying nifi in automated 
> environments such as Kubernetes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFIREG-143) Sort ordering resets after a Flow delete operation

[Andrew Grande (JIRA)]

Andrew Grande created NIFIREG-143:
-

 Summary: Sort ordering resets after a Flow delete operation
 Key: NIFIREG-143
 URL: https://issues.apache.org/jira/browse/NIFIREG-143
 Project: NiFi Registry
  Issue Type: Bug
Affects Versions: 0.1.0
Reporter: Andrew Grande


Display a list of flows in a bucket, order by Newest first. Make sure there are 
several flows listed, at least 3.

Actions -> Delete.

Expected: the ordering is preserved.

Actual: the ordering resets and can potentially cause problems when a user 
tries to perform a semi-bulk operation like deleting multiple commonly named 
flows (e.g. the Quick Import -  ones).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-1023) Need ability to supply namespace mapping for XPath/XQuery processors

[Seth Kingsley (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363426#comment-16363426
 ] 

Seth Kingsley commented on NIFI-1023:
-

I decided to try a quick-and-dirty version of this, using attributes to define 
namespace fixes. For instance, if you set xmlns.ns1 to [http://namespace/1,] 
that makes it available in XPath expressions.

How strong is the requirement to have a controller service keeping track of the 
namespaces instead?

> Need ability to supply namespace mapping for XPath/XQuery processors
> 
>
> Key: NIFI-1023
> URL: https://issues.apache.org/jira/browse/NIFI-1023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Gilman
>Priority: Minor
>
> Possibly consider creating a controller service that provides access to a 
> NamespaceContext which can be used with EvaluteXPath and EvaluateXQuery.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4839) Create a CLI in NiFi Toolkit to interact with NIFi Registry/deploy flows

[Daniel Oakley (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363231#comment-16363231
 ] 

Daniel Oakley commented on NIFI-4839:
-

I had some questions about the intended capabilities of this script:

1) Can it be used to configure a fresh Nifi instance to integrate with a 
nifi-registry?

2) Can it tell Nifi to add a flow from the registry to the base flow?

3) Can it stop/start the base flow and everything in it?

I require the above capabilities in order to automate the deployment of a fresh 
Nifi docker instance and have it load and start a given flow from the registry. 
Currently I embed the flow.xml.gz in the Docker image but it would be nice to 
be able to externalize it through the registry.

> Create a CLI in NiFi Toolkit to interact with NIFi Registry/deploy flows
> 
>
> Key: NIFI-4839
> URL: https://issues.apache.org/jira/browse/NIFI-4839
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
>
> Now that we have NiFi Registry and the ability to import/upgrade flows in 
> NiFi, we should offer a command-line tool to interact with these REST 
> end-points. This could part of NiFi Toolkit and would help people potentially 
> automate some of these operations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4875) Import flow from nifi-registry on bootstrap/initial start

[Daniel Oakley (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363232#comment-16363232
 ] 

Daniel Oakley commented on NIFI-4875:
-

[~pvillard] OK I will follow up with the CLI tool.

> Import flow from nifi-registry on bootstrap/initial start
> -
>
> Key: NIFI-4875
> URL: https://issues.apache.org/jira/browse/NIFI-4875
> Project: Apache NiFi
>  Issue Type: Wish
>  Components: Flow Versioning
>Affects Versions: 1.5.0
>Reporter: Daniel Oakley
>Priority: Major
>  Labels: registry
>
> As I user I would like to be able to automate the deployment of a new Nifi 
> instance (e.g. using Docker, Ansible), and have that instance load a specific 
> versioned flow from the nifi-registry and start running it automatically.
> To enable this I would like to be able to specify the nifi-registry URL in 
> nifi properties file, and the flow to load/start via either the properties 
> file or a command line option.
> This will greatly increase the ease of deploying nifi in automated 
> environments such as Kubernetes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363304#comment-16363304
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/2294
  
@MikeThomsen Thank you for the quick updates! All LGTM now, +1. Merging to 
master.


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2294: NIFI-3538 Added DeleteHBaseRow

[ijokarumawak]

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/2294
  
@MikeThomsen Thank you for the quick updates! All LGTM now, +1. Merging to 
master.


---

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363307#comment-16363307
 ] 

ASF subversion and git services commented on NIFI-3538:
---

Commit 143d7e6829c7ac5967dceb0df4a9f7f4456720eb in nifi's branch 
refs/heads/master from [~mike.thomsen]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=143d7e6 ]

NIFI-3538 Added DeleteHBaseRow

This closes #2294.

Signed-off-by: Koji Kawamura 


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363308#comment-16363308
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2294


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-3538) Add DeleteHBaseRow processor

[Koji Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-3538:

Summary: Add DeleteHBaseRow processor  (was: Add DeleteHBase processor(s))

> Add DeleteHBaseRow processor
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-3538) Add DeleteHBaseRow processor

[Koji Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-3538:

   Resolution: Fixed
Fix Version/s: 1.6.0
   Status: Resolved  (was: Patch Available)

> Add DeleteHBaseRow processor
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.6.0
>
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2425: Emit failures array

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2425#discussion_r168057832
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateRecord.java
 ---
@@ -242,11 +279,12 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final boolean allowExtraFields = 
context.getProperty(ALLOW_EXTRA_FIELDS).asBoolean();
 final boolean strictTypeChecking = 
context.getProperty(STRICT_TYPE_CHECKING).asBoolean();
 
-RecordSetWriter validWriter = null;
-RecordSetWriter invalidWriter = null;
 FlowFile validFlowFile = null;
 FlowFile invalidFlowFile = null;
 
+final List validRecords = new LinkedList<>();
--- End diff --

Hi @martin-mucha 
Let me try to answer your question. @markap14 will correct me if I'm wrong 
:)


[ValidateRecord.completeFlowFile](https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateRecord.java#L408)
 method calls `writer.finishRecordSet()`, which let the writer to write the 
ending mark of record set, as some record format requires this, e.g. JSON '}' 
or XML '' would be easy to imagine. Actual bytes for record contents had 
been written in advance.

I'd recommend reading [NiFi in depth, Content 
Repository](https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html#content-repository)
 on how NiFi reads/writes FlowFile content in streaming manner without loading 
whole content on heap.

If you're interested in reading code, 
[StandardProcessSession.write](https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java#L2433)
 might be a good starting point for how FlowFile and its OutputStream is 
created. 

And the OutputStream is passed to RecordSetWriter implementations. For 
example, when a processor writes a record, then it is sent to a method of a 
configured RecordSetWriter like this, 

[WriteCSVResult.writeRecord](https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/csv/WriteCSVResult.java#L147).

These RecordSetWriter does not hold contents on heap. They write records in 
streaming manner.

If we create a List and hold `Record` instances, then we keep content on 
heap as `Record` instances which can lead to a OOM.

Hope this helps!


---

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2294


---