[jira] [Commented] (NIFI-4715) ListS3 list duplicate files when incoming file throughput to S3 is high

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297848#comment-16297848
 ] 

Joseph Witt commented on NIFI-4715:
---

[~dmilan77]  ListS3 is annotated with TriggerSerially.  This means it can only 
ever have one thread.  It is designed to be run this way exclusively.  When you 
say run in multi-threaded mode are you saying you're able to have it run with 
more than one thread?  Can you share a screen shot.

It is designed to be single threaded for the listing then the listing results 
can be sent around the cluster via S2S protocol and Fetched in parallel.  This 
List/Fetch pattern is extremely common now for massive scale flows.

Please confirm whether there is a bug or a misunderstanding of how it works.

> ListS3 list  duplicate files when incoming file throughput to S3 is high
> 
>
> Key: NIFI-4715
> URL: https://issues.apache.org/jira/browse/NIFI-4715
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0, 1.3.0, 1.4.0
> Environment: All
>Reporter: Milan Das
>
> ListS3 state is implemented using HashSet. HashSet is not thread safe. When 
> ListS3 operates in multi threaded mode, sometimes it  tries to list  same 
> file from S3 bucket.  Seems like HashSet data is getting corrupted.
> currentKeys = new HashSet<>(); // need to be implemented Thread Safe like 
> currentKeys = //ConcurrentHashMap.newKeySet();



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4715) ListS3 list duplicate files when incoming file throughput to S3 is high

[Milan Das (JIRA)]

Milan Das created NIFI-4715:
---

 Summary: ListS3 list  duplicate files when incoming file 
throughput to S3 is high
 Key: NIFI-4715
 URL: https://issues.apache.org/jira/browse/NIFI-4715
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.4.0, 1.3.0, 1.2.0
 Environment: All
Reporter: Milan Das


ListS3 state is implemented using HashSet. HashSet is not thread safe. When 
ListS3 operates in multi threaded mode, sometimes it  tries to list  same file 
from S3 bucket.  Seems like HashSet data is getting corrupted.

currentKeys = new HashSet<>(); // need to be implemented Thread Safe like 
currentKeys = //ConcurrentHashMap.newKeySet();



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4714) S2S does not work with default configuration if hostname has uppercase

[Koji Kawamura (JIRA)]

Koji Kawamura created NIFI-4714:
---

 Summary: S2S does not work with default configuration if hostname 
has uppercase
 Key: NIFI-4714
 URL: https://issues.apache.org/jira/browse/NIFI-4714
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.5.0
Reporter: Koji Kawamura
Assignee: Koji Kawamura
 Fix For: 1.5.0


NIFI-4632 added hostname retrieved by InetAddress.getLocalHost() to valid 
hostnames. But it does not lowercase the hostname while hostHeaderIsValid 
method matches lowercased hostname against valid hosts. This makes S2S with 
default configuration to fail if hostname contains uppercase character.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297635#comment-16297635
 ] 

ASF GitHub Bot commented on NIFI-4325:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157903183
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-api/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchResponse.java
 ---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import java.util.List;
+import java.util.Map;
+
+public class ElasticSearchResponse {
+private List> hits;
+private Map aggregations;
+private long numberOfHits;
+private int took;
+private boolean timedOut;
+
+public ElasticSearchResponse(List> hits, 
Map aggregations,
+ int numberOfHits, int took, boolean 
timedOut) {
+this.hits = hits;
+this.aggregations = aggregations;
+this.numberOfHits = numberOfHits;
+this.took = took;
+this.timedOut = timedOut;
+}
+
+public Map getAggregations() {
+return aggregations;
+}
+
+public List> getHits() {
+return hits;
+}
+
+public long getNumberOfHits() {
+return numberOfHits;
+}
+
+public boolean isTimedOut() {
+return timedOut;
+}
+
+public int getTook() {
+return took;
+}
+
+@Override
+public String toString() {
+return "ElasticSearchResponse{" +
--- End diff --

Nice!


> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for 
> querying, not the JSON DSL. A new processor is needed that can take a full 
> JSON query and execute it. It should also support aggregation queries in this 
> syntax. A user needs to be able to take a query as-is from Kibana and drop it 
> into NiFi and have it just run.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297634#comment-16297634
 ] 

ASF GitHub Bot commented on NIFI-4325:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157902956
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
 ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = 
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The 
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query 
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support 
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error 
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\" 
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new 
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new 
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new 
PropertyDescriptor.Builder()
+.name("el5-fetch-index")

[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297633#comment-16297633
 ] 

ASF GitHub Bot commented on NIFI-4325:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157903679
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-bundle/nifi-elasticsearch-client-service-nar/src/main/resources/META-INF/LICENSE
 ---
@@ -0,0 +1,357 @@
+
+ Apache License
+   Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other 
modifications
+  represent, as a whole, an original work of authorship. For the 
purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces 
of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright 
owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control 
systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a 

[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297636#comment-16297636
 ] 

ASF GitHub Bot commented on NIFI-4325:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157902546
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
 ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = 
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The 
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
--- End diff --

It looks like you're not using the incoming flow file contents, in which 
case we might relax this constraint to INPUT_ALLOWED, and if no flow file is 
available (you can evaluate EL against a null FF) you can create one, what do 
you think?


> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for 
> querying, not the JSON DSL. A new processor is needed that can take a full 
> JSON query and execute it. It should also support aggregation queries in this 
> syntax. A user needs to be able to take a query as-is from Kibana and drop it 
> into NiFi and have it just run.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297631#comment-16297631
 ] 

ASF GitHub Bot commented on NIFI-4325:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157902648
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
 ---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type"),
+@WritesAttribute(attribute = "aggregation.name", description = "The 
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query 
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support 
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error 
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\" 
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new 
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new 
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new 
PropertyDescriptor.Builder()
+.name("el5-fetch-index")
+.displayName("Index")
 

[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297632#comment-16297632
 ] 

ASF GitHub Bot commented on NIFI-4325:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157902770
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
 ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = 
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The 
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query 
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support 
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error 
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\" 
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new 
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new 
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new 
PropertyDescriptor.Builder()
+.name("el5-fetch-index")

[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297629#comment-16297629
 ] 

ASF GitHub Bot commented on NIFI-4325:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157902719
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
 ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = 
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The 
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query 
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support 
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error 
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\" 
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new 
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new 
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new 
PropertyDescriptor.Builder()
+.name("el5-fetch-index")

[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297630#comment-16297630
 ] 

ASF GitHub Bot commented on NIFI-4325:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157903403
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-bundle/nifi-elasticsearch-client-service-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,332 @@
+nifi-hbase_1_1_2-client-service-nar
--- End diff --

This appears to be a copy from a different NAR ;) The first line should be 
updated, and the rest of the NOTICES below should only be included if we are 
including the component(s) in the artifacts.


> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for 
> querying, not the JSON DSL. A new processor is needed that can take a full 
> JSON query and execute it. It should also support aggregation queries in this 
> syntax. A user needs to be able to take a query as-is from Kibana and drop it 
> into NiFi and have it just run.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157902770
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
 ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = 
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The 
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query 
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support 
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error 
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\" 
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new 
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new 
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new 
PropertyDescriptor.Builder()
+.name("el5-fetch-index")
+.displayName("Index")
+.description("The name of the index to read from")
+.required(true)
+.expressionLanguageSupported(true)
+

[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157903183
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-api/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchResponse.java
 ---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import java.util.List;
+import java.util.Map;
+
+public class ElasticSearchResponse {
+private List> hits;
+private Map aggregations;
+private long numberOfHits;
+private int took;
+private boolean timedOut;
+
+public ElasticSearchResponse(List> hits, 
Map aggregations,
+ int numberOfHits, int took, boolean 
timedOut) {
+this.hits = hits;
+this.aggregations = aggregations;
+this.numberOfHits = numberOfHits;
+this.took = took;
+this.timedOut = timedOut;
+}
+
+public Map getAggregations() {
+return aggregations;
+}
+
+public List> getHits() {
+return hits;
+}
+
+public long getNumberOfHits() {
+return numberOfHits;
+}
+
+public boolean isTimedOut() {
+return timedOut;
+}
+
+public int getTook() {
+return took;
+}
+
+@Override
+public String toString() {
+return "ElasticSearchResponse{" +
--- End diff --

Nice!


---

[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157902956
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
 ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = 
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The 
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query 
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support 
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error 
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\" 
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new 
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new 
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new 
PropertyDescriptor.Builder()
+.name("el5-fetch-index")
+.displayName("Index")
+.description("The name of the index to read from")
+.required(true)
+.expressionLanguageSupported(true)
+

[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157902719
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
 ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = 
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The 
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query 
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support 
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error 
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\" 
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new 
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new 
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new 
PropertyDescriptor.Builder()
+.name("el5-fetch-index")
+.displayName("Index")
+.description("The name of the index to read from")
+.required(true)
+.expressionLanguageSupported(true)
+

[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157903679
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-bundle/nifi-elasticsearch-client-service-nar/src/main/resources/META-INF/LICENSE
 ---
@@ -0,0 +1,357 @@
+
+ Apache License
+   Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other 
modifications
+  represent, as a whole, an original work of authorship. For the 
purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces 
of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright 
owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control 
systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  (except as stated in this section) patent license to make, have made,
+  use, offer to sell, sell, import, and otherwise transfer the 

[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157902648
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
 ---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type"),
+@WritesAttribute(attribute = "aggregation.name", description = "The 
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query 
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support 
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error 
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\" 
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new 
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new 
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new 
PropertyDescriptor.Builder()
+.name("el5-fetch-index")
+.displayName("Index")
+.description("The name of the index to read from")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+

[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157902546
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
 ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = 
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The 
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
--- End diff --

It looks like you're not using the incoming flow file contents, in which 
case we might relax this constraint to INPUT_ALLOWED, and if no flow file is 
available (you can evaluate EL against a null FF) you can create one, what do 
you think?


---

[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157903403
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-bundle/nifi-elasticsearch-client-service-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,332 @@
+nifi-hbase_1_1_2-client-service-nar
--- End diff --

This appears to be a copy from a different NAR ;) The first line should be 
updated, and the rest of the NOTICES below should only be included if we are 
including the component(s) in the artifacts.


---

[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297578#comment-16297578
 ] 

ASF GitHub Bot commented on NIFI-4325:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157902041
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
 ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
--- End diff --

Lines 38 and 42 are unused imports (i.e. CheckStyle violations), but I can 
fix these on final merge if there are no other issues :)


> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for 
> querying, not the JSON DSL. A new processor is needed that can take a full 
> JSON query and execute it. It should also support aggregation queries in this 
> syntax. A user needs to be able to take a query as-is from Kibana and drop it 
> into NiFi and have it just run.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2113#discussion_r157902041
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
 ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
--- End diff --

Lines 38 and 42 are unused imports (i.e. CheckStyle violations), but I can 
fix these on final merge if there are no other issues :)


---

[jira] [Commented] (NIFI-614) Create a JMS ConnectionFactory controller service and refactor JMS processors to use it

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297534#comment-16297534
 ] 

ASF GitHub Bot commented on NIFI-614:
-

Github user mosermw commented on the issue:

https://github.com/apache/nifi/pull/222
  
Since TIBCO libraries don't fall under a license that is compatible with 
the Apache License 2.0, Apache NiFi cannot distribute a solution that is 
specific to TIBCO.  But you should be able to build an extension to Apache NiFi 
that works for you.  Here is an example for how you might approach building a 
NiFi extension for this 
[mosermw/nifi-jms-providers-bundle](https://github.com/mosermw/nifi-jms-providers-bundle).


> Create a JMS ConnectionFactory controller service and refactor JMS processors 
> to use it
> ---
>
> Key: NIFI-614
> URL: https://issues.apache.org/jira/browse/NIFI-614
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.1.0
>Reporter: Michael Moser
>Assignee: Oleg Zhurakousky
>  Labels: JMS
> Fix For: 0.6.0
>
>
> It is difficult to extend the standard JMS processors to use a different JMS 
> provider besides the included ActiveMQ.  Improve the process of adding 
> support for other JMS providers.
> Create a JMS ConnectionFactoryService API in nifi-standard-services.  Create 
> an ActiveMQConnectionFactoryService with the ActiveMQ implementation in 
> nifi-standard-services.  Modify the JMS processors in 
> nifi-standard-processors to use a ConnectionFactoryService to pick the JMS 
> Provider to use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #222: NIFI-614 Added initial support for new style JMS

[mosermw]

Github user mosermw commented on the issue:

https://github.com/apache/nifi/pull/222
  
Since TIBCO libraries don't fall under a license that is compatible with 
the Apache License 2.0, Apache NiFi cannot distribute a solution that is 
specific to TIBCO.  But you should be able to build an extension to Apache NiFi 
that works for you.  Here is an example for how you might approach building a 
NiFi extension for this 
[mosermw/nifi-jms-providers-bundle](https://github.com/mosermw/nifi-jms-providers-bundle).


---

[jira] [Created] (NIFI-4713) Datadog Metrics Alignment

[Robert Batts (JIRA)]

Robert Batts created NIFI-4713:
--

 Summary: Datadog Metrics Alignment
 Key: NIFI-4713
 URL: https://issues.apache.org/jira/browse/NIFI-4713
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Affects Versions: 1.4.0
Reporter: Robert Batts


Metrics that are being fed into Datadog from Nifi do not seem to align to the 
Nifi model. Therefore, I am proposing the following.

# Change the metric names to work better with Datadog
# Become more reliant on tagging
# Allow custom tagging

Currently, metrics are being sent to Datadog in the following format:

..

However, Datadog is more of a reuse a metric name and filter via tagging 
system. So in Datadog, something with a metric name of 
. with a tag of  works better than 
one unique metric per processor (in an event where there is no processorName, 
exclude the tag instead of adding 'flow'). 

Consider the way Datadog does Kafka. The metric kafka.consumer_lag represents 
the current lag of a topic (tag) for a given consumer_group (tag) over all 
partitions (tag). 

For the same moment in time:
kafka.consumer_lag = 5 
kafka.consumer_lag = 7 
kafka.consumer_lag = 22 
kafka.consumer_lag = 19 
kafka.consumer_lag = 2 

If I wanted to know what the current lag was for a given consumer_group on all 
topics, I would include those tags and then sum on the remaining records (which 
would be the across the partitions). 

For the same moment in time:
kafka.consumer_lag = 12 for topic:a and consumer_group:nifi
kafka.consumer_lag = 2 for topic:b and consumer_group:nifi

In a Nifi sense, this could allow you to (for example) have a tag that noted 
this was an aws-sqs pull and aggregate the average number of records being 
pulled across the entire system instead of on a single process.

Additionally, there is room for custom tagging as well. For example: I want to 
be able to aggregate across all Nifi clusters I control. Setting the prefix 
unique for each cluster breaks this aggregation and might not allow me to 
filter properly later if I do not set a prefix. But, if custom tagging was 
allowed, I could set a tag for cluster_name:nifi-1 and then you could have all 
metrics aggregated but be able to filter down to that specific cluster for 
other operations. In my opinion, the easiest way to implement this would be to 
take all non-required attributes from the Datadog controller and use them as 
the custom tags (these attributes should be considered final/static when 
loaded). The attributes are already in Key=Value format, so it should be easy 
enough to switch them over to Key:Value formatting for tagging (once the 
required attributes are removed).

(Most if not all work for this is centered on 
org.apache.nifi.reporting.datadog.DataDogReportingTask)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4712) Allow Nifi to connect to Datadog through a proxy

[Robert Batts (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Batts updated NIFI-4712:
---
Labels: datadog metrics  (was: )

> Allow Nifi to connect to Datadog through a proxy
> 
>
> Key: NIFI-4712
> URL: https://issues.apache.org/jira/browse/NIFI-4712
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Robert Batts
>  Labels: datadog, metrics
>
> Nifi should be able to report metrics to Datadog via direct connection 
> through a proxy. Currently, Nifi attempts a direct connection to Datadog 
> (seen in the org.apache.nifi.reporting.datadog.DDMetricRegistryBuilder class) 
> when a API key that is not "agent" is given. This connection should be 
> allowed to go through a proxy instead of just being forced to use the agent. 
> This should be as easy as a .withProxy() 
> (https://github.com/coursera/metrics-datadog/blob/master/metrics-datadog/src/main/java/org/coursera/metrics/datadog/transport/HttpTransport.java#L83)
>  and possibly adding a proxy host and proxy port attribute similar to what is 
> used in the AWS modules.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4712) Allow Nifi to connect to Datadog through a proxy

[Robert Batts (JIRA)]

Robert Batts created NIFI-4712:
--

 Summary: Allow Nifi to connect to Datadog through a proxy
 Key: NIFI-4712
 URL: https://issues.apache.org/jira/browse/NIFI-4712
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Affects Versions: 1.4.0
Reporter: Robert Batts


Nifi should be able to report metrics to Datadog via direct connection through 
a proxy. Currently, Nifi attempts a direct connection to Datadog (seen in the 
org.apache.nifi.reporting.datadog.DDMetricRegistryBuilder class) when a API key 
that is not "agent" is given. This connection should be allowed to go through a 
proxy instead of just being forced to use the agent. This should be as easy as 
a .withProxy() 
(https://github.com/coursera/metrics-datadog/blob/master/metrics-datadog/src/main/java/org/coursera/metrics/datadog/transport/HttpTransport.java#L83)
 and possibly adding a proxy host and proxy port attribute similar to what is 
used in the AWS modules.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-614) Create a JMS ConnectionFactory controller service and refactor JMS processors to use it

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297172#comment-16297172
 ] 

ASF GitHub Bot commented on NIFI-614:
-

Github user jugi92 commented on the issue:

https://github.com/apache/nifi/pull/222
  
Was the Tibco issue ever followed up on?
I tried to set sSLTrustedCertificate as dynamic property but it gives the 
following error:
```
2017-12-19 17:48:09,788 ERROR [StandardProcessScheduler Thread-7] 
o.a.n.c.s.StandardControllerServiceNode 
JMSConnectionFactoryProvider[id=a7cf3b99-c426-18a4--3dc28240] Failed to 
invoke @OnEnabled method due to java.lang.IllegalStateException: 
java.lang.IllegalStateException: Failed to set property sSLTrustedCertificate: 
{}
java.lang.IllegalStateException: java.lang.IllegalStateException: Failed to 
set property sSLTrustedCertificate
```

Tibco Java Doc is available here:

https://docs.tibco.com/pub/enterprise_message_service/6.3.0-february-2012/doc/html/tib_ems_api_reference/api/javadoc/com/tibco/tibjms/TibjmsConnectionFactory.html#setSSLTrustedCertificate(java.lang.String)


> Create a JMS ConnectionFactory controller service and refactor JMS processors 
> to use it
> ---
>
> Key: NIFI-614
> URL: https://issues.apache.org/jira/browse/NIFI-614
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.1.0
>Reporter: Michael Moser
>Assignee: Oleg Zhurakousky
>  Labels: JMS
> Fix For: 0.6.0
>
>
> It is difficult to extend the standard JMS processors to use a different JMS 
> provider besides the included ActiveMQ.  Improve the process of adding 
> support for other JMS providers.
> Create a JMS ConnectionFactoryService API in nifi-standard-services.  Create 
> an ActiveMQConnectionFactoryService with the ActiveMQ implementation in 
> nifi-standard-services.  Modify the JMS processors in 
> nifi-standard-processors to use a ConnectionFactoryService to pick the JMS 
> Provider to use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #222: NIFI-614 Added initial support for new style JMS

[jugi92]

Github user jugi92 commented on the issue:

https://github.com/apache/nifi/pull/222
  
Was the Tibco issue ever followed up on?
I tried to set sSLTrustedCertificate as dynamic property but it gives the 
following error:
```
2017-12-19 17:48:09,788 ERROR [StandardProcessScheduler Thread-7] 
o.a.n.c.s.StandardControllerServiceNode 
JMSConnectionFactoryProvider[id=a7cf3b99-c426-18a4--3dc28240] Failed to 
invoke @OnEnabled method due to java.lang.IllegalStateException: 
java.lang.IllegalStateException: Failed to set property sSLTrustedCertificate: 
{}
java.lang.IllegalStateException: java.lang.IllegalStateException: Failed to 
set property sSLTrustedCertificate
```

Tibco Java Doc is available here:

https://docs.tibco.com/pub/enterprise_message_service/6.3.0-february-2012/doc/html/tib_ems_api_reference/api/javadoc/com/tibco/tibjms/TibjmsConnectionFactory.html#setSSLTrustedCertificate(java.lang.String)


---

[jira] [Commented] (NIFI-4410) PutElasticsearchHttp needs better error handling and logging

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297164#comment-16297164
 ] 

ASF GitHub Bot commented on NIFI-4410:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2175
  
Failure in the case of not-found-while-deleting is subjective IMO. You can 
look at it as "the processor did not perform the task I issued", or "the 
processor accomplished the end goal of the task (i.e. the document is not there 
after the delete)". It seems ES is choosing the latter, but if a general user 
would expect a "failed delete" along the failure relationship, we can treat 
them as such.


> PutElasticsearchHttp needs better error handling and logging
> 
>
> Key: NIFI-4410
> URL: https://issues.apache.org/jira/browse/NIFI-4410
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Joseph Witt
>Assignee: Matt Burgess
>
> https://github.com/apache/nifi/blob/6b5015e39b4233cf230151fb45bebcb21df03730/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L364-L366
> If it cannot extract the reason text it provides a very generic error and 
> there is nothing else logged.  You get no context as to what went wrong and 
> further the condition doesn't cause yielding or anything so there is just a 
> massive flood of errors in logs that dont' advise the user of the problem.
> We need to make sure the information can be made available to help 
> troubleshoot and we need to cause yielding so that such cases do not cause 
> continuous floods of errors.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2175: NIFI-4410: Improved error handling/logging in PutElasticse...

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2175
  
Failure in the case of not-found-while-deleting is subjective IMO. You can 
look at it as "the processor did not perform the task I issued", or "the 
processor accomplished the end goal of the task (i.e. the document is not there 
after the delete)". It seems ES is choosing the latter, but if a general user 
would expect a "failed delete" along the failure relationship, we can treat 
them as such.


---

[jira] [Commented] (NIFI-4410) PutElasticsearchHttp needs better error handling and logging

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297163#comment-16297163
 ] 

ASF GitHub Bot commented on NIFI-4410:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2175
  
So it is Elasticsearch that is treating "not found while deleting" 
differently than "not found while updating", they return different status codes 
with the "error" message in different fields in the JSON. This PR checks for 
both. I'm waffling on whether to treat them as errors, I think I'd like to keep 
them as not errors (see my comment above).
@MikeThomsen Are you seeing "not-found-updates" as failures, and 
"not-found-deletes" as successful? If so then are you ok with that behavior? 
I'm tending towards keeping it that way.


> PutElasticsearchHttp needs better error handling and logging
> 
>
> Key: NIFI-4410
> URL: https://issues.apache.org/jira/browse/NIFI-4410
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Joseph Witt
>Assignee: Matt Burgess
>
> https://github.com/apache/nifi/blob/6b5015e39b4233cf230151fb45bebcb21df03730/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L364-L366
> If it cannot extract the reason text it provides a very generic error and 
> there is nothing else logged.  You get no context as to what went wrong and 
> further the condition doesn't cause yielding or anything so there is just a 
> massive flood of errors in logs that dont' advise the user of the problem.
> We need to make sure the information can be made available to help 
> troubleshoot and we need to cause yielding so that such cases do not cause 
> continuous floods of errors.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2175: NIFI-4410: Improved error handling/logging in PutElasticse...

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2175
  
So it is Elasticsearch that is treating "not found while deleting" 
differently than "not found while updating", they return different status codes 
with the "error" message in different fields in the JSON. This PR checks for 
both. I'm waffling on whether to treat them as errors, I think I'd like to keep 
them as not errors (see my comment above).
@MikeThomsen Are you seeing "not-found-updates" as failures, and 
"not-found-deletes" as successful? If so then are you ok with that behavior? 
I'm tending towards keeping it that way.


---

[jira] [Commented] (NIFI-4696) Support concurrent tasks in PutHiveStreaming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297054#comment-16297054
 ] 

ASF GitHub Bot commented on NIFI-4696:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2342#discussion_r157808792
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/PutHiveStreaming.java
 ---
@@ -156,7 +158,9 @@
 .displayName("Hive Configuration Resources")
 .description("A file or comma separated list of files which 
contains the Hive configuration (hive-site.xml, e.g.). Without this, Hadoop "
 + "will search the classpath for a 'hive-site.xml' 
file or will revert to a default configuration. Note that to enable 
authentication "
-+ "with Kerberos e.g., the appropriate properties must 
be set in the configuration files. Please see the Hive documentation for more 
details.")
++ "with Kerberos e.g., the appropriate properties must 
be set in the configuration files. Also note that if Max Concurrent Tasks is 
set "
++ "to a number greater than one, the 
'hcatalog.hive.client.cache.disabled' property will be forced to 'true' to 
avoid concurrency issues. "
--- End diff --

Yes good point, will add that


> Support concurrent tasks in PutHiveStreaming
> 
>
> Key: NIFI-4696
> URL: https://issues.apache.org/jira/browse/NIFI-4696
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Currently PutHiveStreaming (PHS) can only support a single task at a time. 
> Before NIFI-4342, that meant each target table would need its own PHS 
> instance, which can be cumbersome with large numbers of tables. After 
> NIFI-4342, Expression Language could be used for SDLC purposes 
> (database/table changes between development and production, e.g.).
> However it would be nice to be able to support at least database/table names 
> using flow file attributes, and also to support multiple tasks to handle them 
> concurrently. Due to the nature of PHS and the Streaming Ingest APIs (and 
> implementation), it is likely not prudent to allow two tasks to write to the 
> same table and partition at the same time.
> I propose adding flow file attribute EL evaluation where prudent, and 
> allowing per-table concurrency in PHS. A thread will attempt to get a lock on 
> a table, and if it cannot, will rollback and return.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2342: NIFI-4696: Add Flowfile attribute EL support and pe...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2342#discussion_r157808792
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/PutHiveStreaming.java
 ---
@@ -156,7 +158,9 @@
 .displayName("Hive Configuration Resources")
 .description("A file or comma separated list of files which 
contains the Hive configuration (hive-site.xml, e.g.). Without this, Hadoop "
 + "will search the classpath for a 'hive-site.xml' 
file or will revert to a default configuration. Note that to enable 
authentication "
-+ "with Kerberos e.g., the appropriate properties must 
be set in the configuration files. Please see the Hive documentation for more 
details.")
++ "with Kerberos e.g., the appropriate properties must 
be set in the configuration files. Also note that if Max Concurrent Tasks is 
set "
++ "to a number greater than one, the 
'hcatalog.hive.client.cache.disabled' property will be forced to 'true' to 
avoid concurrency issues. "
--- End diff --

Yes good point, will add that


---

[jira] [Commented] (NIFI-4402) Add component location in Summary view

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296996#comment-16296996
 ] 

ASF GitHub Bot commented on NIFI-4402:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2195
  
Hey @yuri1969 and @mcgilman - just did a test in a secured env where a user 
only has access to a specific process group. It LGTM:

https://user-images.githubusercontent.com/11541012/34165711-509f4358-e4dd-11e7-8e43-dfad8f667a7f.png;>

Does it sound OK to you @mcgilman ?
Happy to check something else if needed. Thanks!



> Add component location in Summary view
> --
>
> Key: NIFI-4402
> URL: https://issues.apache.org/jira/browse/NIFI-4402
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Pierre Villard
>Priority: Minor
>
> In the summary view, when there are hundreds of processors, it can be 
> difficult to know what exact processor we are looking at in the table. The 
> only options are:
> - go to the processor location to know what processor it is (but then we are 
> not on the summary view anymore)
> - have a naming convention to uniquely name every component
> To improve the user experience, the following could be implemented:
> - add a column "Process Group" displaying the name of the parent process 
> group containing the component
> - when hovering the "Go to location" button change the tooltip to include the 
> path of the component. Something like: "Go to /NiFi Flow/My RPG/My Processor" 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2195: NIFI-4402 - Add component location in Summary view

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2195
  
Hey @yuri1969 and @mcgilman - just did a test in a secured env where a user 
only has access to a specific process group. It LGTM:

https://user-images.githubusercontent.com/11541012/34165711-509f4358-e4dd-11e7-8e43-dfad8f667a7f.png;>

Does it sound OK to you @mcgilman ?
Happy to check something else if needed. Thanks!



---

[jira] [Resolved] (NIFI-4005) Add support for Azure Shared Access Signature (SAS) Tokens

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman resolved NIFI-4005.
---
   Resolution: Fixed
Fix Version/s: 1.5.0

> Add support for Azure Shared Access Signature (SAS) Tokens
> --
>
> Key: NIFI-4005
> URL: https://issues.apache.org/jira/browse/NIFI-4005
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Andrew Grande
>Assignee: Koji Kawamura
>Priority: Minor
> Fix For: 1.5.0
>
>
> Azure's account name and key are treated as admin, 'root' access credentials. 
> If one has those, every container under this account is fully accessible. An 
> MSFT-recommended approach is to use SAS policies, which provide for a fine 
> grained permission and object control, as well as defined expiration.
> I already have working code, filing this ticket to formally track and submit 
> PR against next.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4005) Add support for Azure Shared Access Signature (SAS) Tokens

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296976#comment-16296976
 ] 

ASF subversion and git services commented on NIFI-4005:
---

Commit 1ee8d16a211c247d1d3d7f1189be36a866d0ee98 in nifi's branch 
refs/heads/master from [~ijokarumawak]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=1ee8d16 ]

NIFI-4005: Azure Blob Storage SAS support, incorporating review comments. This 
closes #2353

- Renamed Azure to AzureStorageUtils.
- Fixed whitespacing in property description.
- Renamed SAS String to SAS Token.


> Add support for Azure Shared Access Signature (SAS) Tokens
> --
>
> Key: NIFI-4005
> URL: https://issues.apache.org/jira/browse/NIFI-4005
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Andrew Grande
>Assignee: Koji Kawamura
>Priority: Minor
>
> Azure's account name and key are treated as admin, 'root' access credentials. 
> If one has those, every container under this account is fully accessible. An 
> MSFT-recommended approach is to use SAS policies, which provide for a fine 
> grained permission and object control, as well as defined expiration.
> I already have working code, filing this ticket to formally track and submit 
> PR against next.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4005) Add support for Azure Shared Access Signature (SAS) Tokens

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296978#comment-16296978
 ] 

ASF GitHub Bot commented on NIFI-4005:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2353
  
Thanks @ijokarumawak @aperepel! This has been merged to master.


> Add support for Azure Shared Access Signature (SAS) Tokens
> --
>
> Key: NIFI-4005
> URL: https://issues.apache.org/jira/browse/NIFI-4005
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Andrew Grande
>Assignee: Koji Kawamura
>Priority: Minor
>
> Azure's account name and key are treated as admin, 'root' access credentials. 
> If one has those, every container under this account is fully accessible. An 
> MSFT-recommended approach is to use SAS policies, which provide for a fine 
> grained permission and object control, as well as defined expiration.
> I already have working code, filing this ticket to formally track and submit 
> PR against next.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4005) Add support for Azure Shared Access Signature (SAS) Tokens

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296979#comment-16296979
 ] 

ASF GitHub Bot commented on NIFI-4005:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2353


> Add support for Azure Shared Access Signature (SAS) Tokens
> --
>
> Key: NIFI-4005
> URL: https://issues.apache.org/jira/browse/NIFI-4005
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Andrew Grande
>Assignee: Koji Kawamura
>Priority: Minor
>
> Azure's account name and key are treated as admin, 'root' access credentials. 
> If one has those, every container under this account is fully accessible. An 
> MSFT-recommended approach is to use SAS policies, which provide for a fine 
> grained permission and object control, as well as defined expiration.
> I already have working code, filing this ticket to formally track and submit 
> PR against next.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4005) Add support for Azure Shared Access Signature (SAS) Tokens

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296975#comment-16296975
 ] 

ASF subversion and git services commented on NIFI-4005:
---

Commit 17ddaf6be0d4c75aa8b0d86f84f1f90757f8368f in nifi's branch 
refs/heads/master from [~aperepel]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=17ddaf6 ]

NIFI-4005 Add support for Azure Shared Access Signature (SAS) Tokens. Upgraded 
the client library from 5.0.0 to 5.2.0


> Add support for Azure Shared Access Signature (SAS) Tokens
> --
>
> Key: NIFI-4005
> URL: https://issues.apache.org/jira/browse/NIFI-4005
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Andrew Grande
>Assignee: Koji Kawamura
>Priority: Minor
>
> Azure's account name and key are treated as admin, 'root' access credentials. 
> If one has those, every container under this account is fully accessible. An 
> MSFT-recommended approach is to use SAS policies, which provide for a fine 
> grained permission and object control, as well as defined expiration.
> I already have working code, filing this ticket to formally track and submit 
> PR against next.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2353: NIFI-4005: Add support for Azure Shared Access Sign...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2353


---

[GitHub] nifi issue #2353: NIFI-4005: Add support for Azure Shared Access Signature (...

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2353
  
Thanks @ijokarumawak @aperepel! This has been merged to master.


---

[jira] [Commented] (NIFI-4005) Add support for Azure Shared Access Signature (SAS) Tokens

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296966#comment-16296966
 ] 

ASF GitHub Bot commented on NIFI-4005:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2353
  
Will review...


> Add support for Azure Shared Access Signature (SAS) Tokens
> --
>
> Key: NIFI-4005
> URL: https://issues.apache.org/jira/browse/NIFI-4005
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Andrew Grande
>Assignee: Koji Kawamura
>Priority: Minor
>
> Azure's account name and key are treated as admin, 'root' access credentials. 
> If one has those, every container under this account is fully accessible. An 
> MSFT-recommended approach is to use SAS policies, which provide for a fine 
> grained permission and object control, as well as defined expiration.
> I already have working code, filing this ticket to formally track and submit 
> PR against next.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2353: NIFI-4005: Add support for Azure Shared Access Signature (...

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2353
  
Will review...


---

[jira] [Resolved] (MINIFICPP-328) Add user docs for MiNiFi - C++ processors

[Andrew Christianson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Christianson resolved MINIFICPP-328.
---
Resolution: Fixed

> Add user docs for MiNiFi - C++ processors
> -
>
> Key: MINIFICPP-328
> URL: https://issues.apache.org/jira/browse/MINIFICPP-328
> Project: NiFi MiNiFi C++
>  Issue Type: Documentation
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>
> Because MiNiFi - C++ processors are currently different from what's offered 
> in NiFi, and will likely remain different to some degree, we can't continue 
> to rely solely on NiFi's documentation. We need to improve MiNiFi's 
> documentation to cover all processors included in MiNiFi and what 
> features/options they support, how they work, etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MINIFICPP-346) Add bustache to bootstrap

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296935#comment-16296935
 ] 

ASF GitHub Bot commented on MINIFICPP-346:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/222


> Add bustache to bootstrap
> -
>
> Key: MINIFICPP-346
> URL: https://issues.apache.org/jira/browse/MINIFICPP-346
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: marco polo
>Assignee: marco polo
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (MINIFICPP-330) Implement substring operations in Expression Language

[Andrew Christianson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Christianson resolved MINIFICPP-330.
---
Resolution: Fixed

> Implement substring operations in Expression Language
> -
>
> Key: MINIFICPP-330
> URL: https://issues.apache.org/jira/browse/MINIFICPP-330
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>
> Add support for these substring functions to EL:
> substring
> substringBefore
> substringBeforeLast
> substringAfter
> substringAfterLast



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (MINIFICPP-331) Implement string replacement functions in Expression Language

[Andrew Christianson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Christianson reassigned MINIFICPP-331:
-

Assignee: Andrew Christianson

> Implement string replacement functions in Expression Language
> -
>
> Key: MINIFICPP-331
> URL: https://issues.apache.org/jira/browse/MINIFICPP-331
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>
> Implement these in EL:
> replace
> replaceFirst
> replaceAll
> replaceNull
> replaceEmpty



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-minifi-cpp pull request #222: MINIFICPP-346: Add raspi distro

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/222


---

[jira] [Commented] (NIFI-4539) ExtractGrok - Add support returning only named captures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296903#comment-16296903
 ] 

ASF GitHub Bot commented on NIFI-4539:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2233


> ExtractGrok - Add support returning only named captures
> ---
>
> Key: NIFI-4539
> URL: https://issues.apache.org/jira/browse/NIFI-4539
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Norito Agetsuma
> Fix For: 1.5.0
>
>
> ExtractGrok support named captures only option.
> Currently, ExtractGrok returns all matches for a grok pattern. In some case, 
> this is verbose.
> Following example parse apache common access log.
> {noformat}
> 83.149.9.216 - - [17/May/2015:10:05:03 +] "GET 
> /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 
> 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/; 
> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, 
> like Gecko) Chrome/32.0.1700.77 Safari/537.36
> {noformat}
> Disable named captures only
> {noformat}
> {
>   "grok.auth": "-",
>   "grok.timestamp": "17/May/2015:10:05:03 +",
>   "grok.httpversion": "1.1",
>   "grok.HOUR": "10",
>   "grok.ident": "-",
>   "grok.SECOND": "03",
>   "grok.HTTPD_COMMONLOG": "83.149.9.216 - - [17/May/2015:10:05:03 +] 
> \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png 
> HTTP/1.1\" 200 203023",
>   "grok.USERNAME": "[-, -]",
>   "grok.IP": "83.149.9.216",
>   "grok.clientip": "83.149.9.216",
>   "grok.verb": "GET",
>   "grok.EMAILADDRESS": "[null, null]",
>   "grok.request": 
> "/presentations/logstash-monitorama-2013/images/kibana-search.png",
>   "grok.EMAILLOCALPART": "[null, null]",
>   "grok.INT": "+",
>   "grok.BASE10NUM": "[1.1, 200, 203023]",
>   "grok.YEAR": "2015",
>   "grok.IPV4": "83.149.9.216",
>   "grok.MINUTE": "05",
>   "grok.HOSTNAME": "[null, null, null]",
>   "grok.USER": "[-, -]",
>   "grok.response": "200",
>   "grok.bytes": "203023",
>   "grok.TIME": "10:05:03",
>   "grok.MONTH": "May",
>   "grok.MONTHDAY": "17"
> }
> {noformat}
> Enable named captures only
> {noformat}
> {
>   "grok.request": 
> "/presentations/logstash-monitorama-2013/images/kibana-search.png",
>   "grok.auth": "-",
>   "grok.ident": "-",
>   "grok.timestamp": "17/May/2015:10:05:03 +",
>   "grok.httpversion": "1.1",
>   "grok.clientip": "83.149.9.216",
>   "grok.response": "200",
>   "grok.bytes": "203023",
>   "grok.verb": "GET"
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4539) ExtractGrok - Add support returning only named captures

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4539:
-
   Resolution: Fixed
Fix Version/s: 1.5.0
   Status: Resolved  (was: Patch Available)

> ExtractGrok - Add support returning only named captures
> ---
>
> Key: NIFI-4539
> URL: https://issues.apache.org/jira/browse/NIFI-4539
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Norito Agetsuma
> Fix For: 1.5.0
>
>
> ExtractGrok support named captures only option.
> Currently, ExtractGrok returns all matches for a grok pattern. In some case, 
> this is verbose.
> Following example parse apache common access log.
> {noformat}
> 83.149.9.216 - - [17/May/2015:10:05:03 +] "GET 
> /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 
> 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/; 
> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, 
> like Gecko) Chrome/32.0.1700.77 Safari/537.36
> {noformat}
> Disable named captures only
> {noformat}
> {
>   "grok.auth": "-",
>   "grok.timestamp": "17/May/2015:10:05:03 +",
>   "grok.httpversion": "1.1",
>   "grok.HOUR": "10",
>   "grok.ident": "-",
>   "grok.SECOND": "03",
>   "grok.HTTPD_COMMONLOG": "83.149.9.216 - - [17/May/2015:10:05:03 +] 
> \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png 
> HTTP/1.1\" 200 203023",
>   "grok.USERNAME": "[-, -]",
>   "grok.IP": "83.149.9.216",
>   "grok.clientip": "83.149.9.216",
>   "grok.verb": "GET",
>   "grok.EMAILADDRESS": "[null, null]",
>   "grok.request": 
> "/presentations/logstash-monitorama-2013/images/kibana-search.png",
>   "grok.EMAILLOCALPART": "[null, null]",
>   "grok.INT": "+",
>   "grok.BASE10NUM": "[1.1, 200, 203023]",
>   "grok.YEAR": "2015",
>   "grok.IPV4": "83.149.9.216",
>   "grok.MINUTE": "05",
>   "grok.HOSTNAME": "[null, null, null]",
>   "grok.USER": "[-, -]",
>   "grok.response": "200",
>   "grok.bytes": "203023",
>   "grok.TIME": "10:05:03",
>   "grok.MONTH": "May",
>   "grok.MONTHDAY": "17"
> }
> {noformat}
> Enable named captures only
> {noformat}
> {
>   "grok.request": 
> "/presentations/logstash-monitorama-2013/images/kibana-search.png",
>   "grok.auth": "-",
>   "grok.ident": "-",
>   "grok.timestamp": "17/May/2015:10:05:03 +",
>   "grok.httpversion": "1.1",
>   "grok.clientip": "83.149.9.216",
>   "grok.response": "200",
>   "grok.bytes": "203023",
>   "grok.verb": "GET"
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4539) ExtractGrok - Add support returning only named captures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296901#comment-16296901
 ] 

ASF GitHub Bot commented on NIFI-4539:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2233
  
Code LGTM, confirmed expected behavior when running simple tests. Thanks 
for your contribution @n-agetsu ! Merging to master...


> ExtractGrok - Add support returning only named captures
> ---
>
> Key: NIFI-4539
> URL: https://issues.apache.org/jira/browse/NIFI-4539
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Norito Agetsuma
>
> ExtractGrok support named captures only option.
> Currently, ExtractGrok returns all matches for a grok pattern. In some case, 
> this is verbose.
> Following example parse apache common access log.
> {noformat}
> 83.149.9.216 - - [17/May/2015:10:05:03 +] "GET 
> /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 
> 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/; 
> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, 
> like Gecko) Chrome/32.0.1700.77 Safari/537.36
> {noformat}
> Disable named captures only
> {noformat}
> {
>   "grok.auth": "-",
>   "grok.timestamp": "17/May/2015:10:05:03 +",
>   "grok.httpversion": "1.1",
>   "grok.HOUR": "10",
>   "grok.ident": "-",
>   "grok.SECOND": "03",
>   "grok.HTTPD_COMMONLOG": "83.149.9.216 - - [17/May/2015:10:05:03 +] 
> \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png 
> HTTP/1.1\" 200 203023",
>   "grok.USERNAME": "[-, -]",
>   "grok.IP": "83.149.9.216",
>   "grok.clientip": "83.149.9.216",
>   "grok.verb": "GET",
>   "grok.EMAILADDRESS": "[null, null]",
>   "grok.request": 
> "/presentations/logstash-monitorama-2013/images/kibana-search.png",
>   "grok.EMAILLOCALPART": "[null, null]",
>   "grok.INT": "+",
>   "grok.BASE10NUM": "[1.1, 200, 203023]",
>   "grok.YEAR": "2015",
>   "grok.IPV4": "83.149.9.216",
>   "grok.MINUTE": "05",
>   "grok.HOSTNAME": "[null, null, null]",
>   "grok.USER": "[-, -]",
>   "grok.response": "200",
>   "grok.bytes": "203023",
>   "grok.TIME": "10:05:03",
>   "grok.MONTH": "May",
>   "grok.MONTHDAY": "17"
> }
> {noformat}
> Enable named captures only
> {noformat}
> {
>   "grok.request": 
> "/presentations/logstash-monitorama-2013/images/kibana-search.png",
>   "grok.auth": "-",
>   "grok.ident": "-",
>   "grok.timestamp": "17/May/2015:10:05:03 +",
>   "grok.httpversion": "1.1",
>   "grok.clientip": "83.149.9.216",
>   "grok.response": "200",
>   "grok.bytes": "203023",
>   "grok.verb": "GET"
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2233: NIFI-4539 ExtractGrok - Add support returning only ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2233


---

[jira] [Commented] (NIFI-4539) ExtractGrok - Add support returning only named captures

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296902#comment-16296902
 ] 

ASF subversion and git services commented on NIFI-4539:
---

Commit f04ddcf44294db64738cb7505c488b83d52c9135 in nifi's branch 
refs/heads/master from [~Norito Agetsuma]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=f04ddcf ]

NIFI-4539 - ExtractGrok - Add support returning only named captures

Signed-off-by: Pierre Villard 

This closes #2233.


> ExtractGrok - Add support returning only named captures
> ---
>
> Key: NIFI-4539
> URL: https://issues.apache.org/jira/browse/NIFI-4539
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Norito Agetsuma
>
> ExtractGrok support named captures only option.
> Currently, ExtractGrok returns all matches for a grok pattern. In some case, 
> this is verbose.
> Following example parse apache common access log.
> {noformat}
> 83.149.9.216 - - [17/May/2015:10:05:03 +] "GET 
> /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 
> 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/; 
> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, 
> like Gecko) Chrome/32.0.1700.77 Safari/537.36
> {noformat}
> Disable named captures only
> {noformat}
> {
>   "grok.auth": "-",
>   "grok.timestamp": "17/May/2015:10:05:03 +",
>   "grok.httpversion": "1.1",
>   "grok.HOUR": "10",
>   "grok.ident": "-",
>   "grok.SECOND": "03",
>   "grok.HTTPD_COMMONLOG": "83.149.9.216 - - [17/May/2015:10:05:03 +] 
> \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png 
> HTTP/1.1\" 200 203023",
>   "grok.USERNAME": "[-, -]",
>   "grok.IP": "83.149.9.216",
>   "grok.clientip": "83.149.9.216",
>   "grok.verb": "GET",
>   "grok.EMAILADDRESS": "[null, null]",
>   "grok.request": 
> "/presentations/logstash-monitorama-2013/images/kibana-search.png",
>   "grok.EMAILLOCALPART": "[null, null]",
>   "grok.INT": "+",
>   "grok.BASE10NUM": "[1.1, 200, 203023]",
>   "grok.YEAR": "2015",
>   "grok.IPV4": "83.149.9.216",
>   "grok.MINUTE": "05",
>   "grok.HOSTNAME": "[null, null, null]",
>   "grok.USER": "[-, -]",
>   "grok.response": "200",
>   "grok.bytes": "203023",
>   "grok.TIME": "10:05:03",
>   "grok.MONTH": "May",
>   "grok.MONTHDAY": "17"
> }
> {noformat}
> Enable named captures only
> {noformat}
> {
>   "grok.request": 
> "/presentations/logstash-monitorama-2013/images/kibana-search.png",
>   "grok.auth": "-",
>   "grok.ident": "-",
>   "grok.timestamp": "17/May/2015:10:05:03 +",
>   "grok.httpversion": "1.1",
>   "grok.clientip": "83.149.9.216",
>   "grok.response": "200",
>   "grok.bytes": "203023",
>   "grok.verb": "GET"
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2233: NIFI-4539 ExtractGrok - Add support returning only named c...

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2233
  
Code LGTM, confirmed expected behavior when running simple tests. Thanks 
for your contribution @n-agetsu ! Merging to master...


---

[GitHub] nifi pull request #2351: NIFI-4707: S2SProvenanceReportingTask not returning...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2351#discussion_r157756243
  
--- Diff: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ---
@@ -640,7 +640,7 @@ private void 
consumeNiFiProvenanceEvents(ReportingContext context, NiFiFlow nifi
 final AnalysisContext analysisContext = new 
StandardAnalysisContext(nifiFlow, clusterResolvers,
 // FIXME: This class cast shouldn't be necessary to query 
lineage. Possible refactor target in next major update.
 
(ProvenanceRepository)eventAccess.getProvenanceRepository());
-consumer.consumeEvents(eventAccess, context.getStateManager(), 
events -> {
+consumer.consumeEvents(null, eventAccess, 
context.getStateManager(), events -> {
--- End diff --

Agreed, I will update


---

[jira] [Commented] (NIFI-4707) SiteToSiteProvenanceReportingTask not returning correct metadata

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296797#comment-16296797
 ] 

ASF GitHub Bot commented on NIFI-4707:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2351#discussion_r157756093
  
--- Diff: 
nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteProvenanceReportingTask.java
 ---
@@ -174,34 +177,47 @@ public void onUnscheduled() {
 return properties;
 }
 
-private Map createComponentMap(final ProcessGroupStatus 
status) {
-final Map componentMap = new HashMap<>();
+private ComponentMapHolder createComponentMap(final ProcessGroupStatus 
status) {
--- End diff --

I agree, will move it there.


> SiteToSiteProvenanceReportingTask not returning correct metadata
> 
>
> Key: NIFI-4707
> URL: https://issues.apache.org/jira/browse/NIFI-4707
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> When the SiteToSiteProvenanceReportingTask emits flow files, some of them 
> include a "componentName" field and some do not. Investigation shows that 
> only the components (except connections) in the root process group have that 
> field populated. Having this information can be very helpful to the user, 
> even though the names might be duplicated, there would be a mapping between a 
> component's ID and its name. At the very least the behavior (i.e. component 
> name being available) should be consistent.
> Having a full map (by traversing the entire flow) also opens up the ability 
> to include Process Group information for the various components. The 
> reporting task could include the parent Process Group identifier and/or name, 
> with perhaps a special ID for the root PG's "parent", such as "@ROOT@" or 
> something unique.
> This could also allow for a PG ID in the list of filtered "component IDs", 
> where any provenance event for a processor in a particular PG could be 
> included in a filter when that PG's ID is in the filter list.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4707) SiteToSiteProvenanceReportingTask not returning correct metadata

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296798#comment-16296798
 ] 

ASF GitHub Bot commented on NIFI-4707:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2351#discussion_r157756243
  
--- Diff: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ---
@@ -640,7 +640,7 @@ private void 
consumeNiFiProvenanceEvents(ReportingContext context, NiFiFlow nifi
 final AnalysisContext analysisContext = new 
StandardAnalysisContext(nifiFlow, clusterResolvers,
 // FIXME: This class cast shouldn't be necessary to query 
lineage. Possible refactor target in next major update.
 
(ProvenanceRepository)eventAccess.getProvenanceRepository());
-consumer.consumeEvents(eventAccess, context.getStateManager(), 
events -> {
+consumer.consumeEvents(null, eventAccess, 
context.getStateManager(), events -> {
--- End diff --

Agreed, I will update


> SiteToSiteProvenanceReportingTask not returning correct metadata
> 
>
> Key: NIFI-4707
> URL: https://issues.apache.org/jira/browse/NIFI-4707
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> When the SiteToSiteProvenanceReportingTask emits flow files, some of them 
> include a "componentName" field and some do not. Investigation shows that 
> only the components (except connections) in the root process group have that 
> field populated. Having this information can be very helpful to the user, 
> even though the names might be duplicated, there would be a mapping between a 
> component's ID and its name. At the very least the behavior (i.e. component 
> name being available) should be consistent.
> Having a full map (by traversing the entire flow) also opens up the ability 
> to include Process Group information for the various components. The 
> reporting task could include the parent Process Group identifier and/or name, 
> with perhaps a special ID for the root PG's "parent", such as "@ROOT@" or 
> something unique.
> This could also allow for a PG ID in the list of filtered "component IDs", 
> where any provenance event for a processor in a particular PG could be 
> included in a filter when that PG's ID is in the filter list.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2251: NIFI-4424 org.apache.nifi.NiFi does not allow programmatic...

[peter-gergely-horvath]

Github user peter-gergely-horvath commented on the issue:

https://github.com/apache/nifi/pull/2251
  
Hi @patricker, please let me know if there is anything that requires 
clarification or if there is any way I could help the review. I would love to 
see this pull request being merged.



---

[jira] [Commented] (NIFI-4005) Add support for Azure Shared Access Signature (SAS) Tokens

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296736#comment-16296736
 ] 

ASF GitHub Bot commented on NIFI-4005:
--

Github user aperepel closed the pull request at:

https://github.com/apache/nifi/pull/1886


> Add support for Azure Shared Access Signature (SAS) Tokens
> --
>
> Key: NIFI-4005
> URL: https://issues.apache.org/jira/browse/NIFI-4005
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Andrew Grande
>Assignee: Koji Kawamura
>Priority: Minor
>
> Azure's account name and key are treated as admin, 'root' access credentials. 
> If one has those, every container under this account is fully accessible. An 
> MSFT-recommended approach is to use SAS policies, which provide for a fine 
> grained permission and object control, as well as defined expiration.
> I already have working code, filing this ticket to formally track and submit 
> PR against next.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4005) Add support for Azure Shared Access Signature (SAS) Tokens

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296735#comment-16296735
 ] 

ASF GitHub Bot commented on NIFI-4005:
--

Github user aperepel commented on the issue:

https://github.com/apache/nifi/pull/1886
  
Thank you, Koji!


> Add support for Azure Shared Access Signature (SAS) Tokens
> --
>
> Key: NIFI-4005
> URL: https://issues.apache.org/jira/browse/NIFI-4005
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Andrew Grande
>Assignee: Koji Kawamura
>Priority: Minor
>
> Azure's account name and key are treated as admin, 'root' access credentials. 
> If one has those, every container under this account is fully accessible. An 
> MSFT-recommended approach is to use SAS policies, which provide for a fine 
> grained permission and object control, as well as defined expiration.
> I already have working code, filing this ticket to formally track and submit 
> PR against next.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1886: NIFI-4005 Add support for Azure Shared Access Signature (S...

[aperepel]

Github user aperepel commented on the issue:

https://github.com/apache/nifi/pull/1886
  
Thank you, Koji!


---

[GitHub] nifi pull request #1886: NIFI-4005 Add support for Azure Shared Access Signa...

[aperepel]

Github user aperepel closed the pull request at:

https://github.com/apache/nifi/pull/1886


---

[jira] [Commented] (NIFI-1490) Add multipart request support to ListenHTTP Processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296612#comment-16296612
 ] 

ASF GitHub Bot commented on NIFI-1490:
--

Github user pipern commented on the issue:

https://github.com/apache/nifi/pull/1795
  
Do you think it's worth it @joewitt , since maybe this PR should be 
rejected and someone works on ExtractMultiPartHttp instead? (I'd like to, but 
can't commit to it. Maybe over Christmas...)

The decision of ExtractMultiPartHttp vs. an extracting-ListenHTTP maybe 
comes down to how we want errors send back to the client. If we want to be able 
to tell the HTTP client that their MultiPart was not well-formed, we'd have to 
do it in ListenHTTP, right? (Which potentially invalidates my first paragraph.)


> Add multipart request support to ListenHTTP Processor
> -
>
> Key: NIFI-1490
> URL: https://issues.apache.org/jira/browse/NIFI-1490
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andrew Serff
>
> The current ListenHTTP processor does not seem to support multipart requests 
> that are encoded with multipart/form-data.  When a multipart request is 
> received, the ListenHTTPServlet just copies the Request InputStream to the 
> FlowFiles content which leaves the form encoding wrapper in the content and 
> in turn makes the file invalid. 
> Specifically, we want to be able to support file uploads in a multipart 
> request. 
> See this thread in the mailing list for more info: 
> http://mail-archives.apache.org/mod_mbox/nifi-users/201602.mbox/%3C6DE9CEEF-2A37-480F-8D3C-5028C590FD9E%40acesinc.net%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1795: NIFI-1490 Add multipart request support to ListenHTTP Proc...

[pipern]

Github user pipern commented on the issue:

https://github.com/apache/nifi/pull/1795
  
Do you think it's worth it @joewitt , since maybe this PR should be 
rejected and someone works on ExtractMultiPartHttp instead? (I'd like to, but 
can't commit to it. Maybe over Christmas...)

The decision of ExtractMultiPartHttp vs. an extracting-ListenHTTP maybe 
comes down to how we want errors send back to the client. If we want to be able 
to tell the HTTP client that their MultiPart was not well-formed, we'd have to 
do it in ListenHTTP, right? (Which potentially invalidates my first paragraph.)


---

[jira] [Commented] (NIFI-4428) Implement PutDruid Processor and Controller

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296557#comment-16296557
 ] 

ASF GitHub Bot commented on NIFI-4428:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2310#discussion_r157704512
  
--- Diff: 
nifi-nar-bundles/nifi-druid-bundle/nifi-druid-controller-service/src/main/java/org/apache/nifi/controller/druid/DruidTranquilityController.java
 ---
@@ -0,0 +1,614 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.druid;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.api.druid.DruidTranquilityService;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.codehaus.jackson.map.ObjectMapper;
+
+import com.metamx.common.Granularity;
+import com.metamx.tranquility.beam.Beam;
+import com.metamx.tranquility.beam.ClusteredBeamTuning;
+import com.metamx.tranquility.druid.DruidBeamConfig;
+import com.metamx.tranquility.druid.DruidBeams;
+import com.metamx.tranquility.druid.DruidDimensions;
+import com.metamx.tranquility.druid.DruidEnvironment;
+import com.metamx.tranquility.druid.DruidLocation;
+import com.metamx.tranquility.druid.DruidRollup;
+import com.metamx.tranquility.tranquilizer.Tranquilizer;
+import com.metamx.tranquility.typeclass.Timestamper;
+
+import io.druid.data.input.impl.TimestampSpec;
+import io.druid.granularity.QueryGranularity;
+import io.druid.query.aggregation.AggregatorFactory;
+import io.druid.query.aggregation.CountAggregatorFactory;
+import io.druid.query.aggregation.DoubleMaxAggregatorFactory;
+import io.druid.query.aggregation.DoubleMinAggregatorFactory;
+import io.druid.query.aggregation.DoubleSumAggregatorFactory;
+import io.druid.query.aggregation.LongMaxAggregatorFactory;
+import io.druid.query.aggregation.LongMinAggregatorFactory;
+import io.druid.query.aggregation.LongSumAggregatorFactory;
+import org.joda.time.DateTime;
+import org.joda.time.Period;
+
+
+@Tags({"Druid", "Timeseries", "OLAP", "ingest"})
+@CapabilityDescription("Asynchronously sends flowfiles to Druid Indexing 
Task using Tranquility API. "
++ "If aggregation and roll-up of data is required, an Aggregator 
JSON descriptor needs to be provided."
++ "Details on how describe aggregation using JSON can be found at: 
http://druid.io/docs/latest/querying/aggregations.html;)
+public class DruidTranquilityController extends AbstractControllerService 
implements DruidTranquilityService {
+private final static String FIREHOSE_PATTERN = "druid:firehose:%s";
+
+private final static AllowableValue PT1M = new AllowableValue("PT1M", 
"1 minute", "1 minute");
+private 

[GitHub] nifi pull request #2310: NIFI-4428: Add PutDruidRecord processor and DruidTr...

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2310#discussion_r157704512
  
--- Diff: 
nifi-nar-bundles/nifi-druid-bundle/nifi-druid-controller-service/src/main/java/org/apache/nifi/controller/druid/DruidTranquilityController.java
 ---
@@ -0,0 +1,614 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.druid;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.api.druid.DruidTranquilityService;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.codehaus.jackson.map.ObjectMapper;
+
+import com.metamx.common.Granularity;
+import com.metamx.tranquility.beam.Beam;
+import com.metamx.tranquility.beam.ClusteredBeamTuning;
+import com.metamx.tranquility.druid.DruidBeamConfig;
+import com.metamx.tranquility.druid.DruidBeams;
+import com.metamx.tranquility.druid.DruidDimensions;
+import com.metamx.tranquility.druid.DruidEnvironment;
+import com.metamx.tranquility.druid.DruidLocation;
+import com.metamx.tranquility.druid.DruidRollup;
+import com.metamx.tranquility.tranquilizer.Tranquilizer;
+import com.metamx.tranquility.typeclass.Timestamper;
+
+import io.druid.data.input.impl.TimestampSpec;
+import io.druid.granularity.QueryGranularity;
+import io.druid.query.aggregation.AggregatorFactory;
+import io.druid.query.aggregation.CountAggregatorFactory;
+import io.druid.query.aggregation.DoubleMaxAggregatorFactory;
+import io.druid.query.aggregation.DoubleMinAggregatorFactory;
+import io.druid.query.aggregation.DoubleSumAggregatorFactory;
+import io.druid.query.aggregation.LongMaxAggregatorFactory;
+import io.druid.query.aggregation.LongMinAggregatorFactory;
+import io.druid.query.aggregation.LongSumAggregatorFactory;
+import org.joda.time.DateTime;
+import org.joda.time.Period;
+
+
+@Tags({"Druid", "Timeseries", "OLAP", "ingest"})
+@CapabilityDescription("Asynchronously sends flowfiles to Druid Indexing 
Task using Tranquility API. "
++ "If aggregation and roll-up of data is required, an Aggregator 
JSON descriptor needs to be provided."
++ "Details on how describe aggregation using JSON can be found at: 
http://druid.io/docs/latest/querying/aggregations.html;)
+public class DruidTranquilityController extends AbstractControllerService 
implements DruidTranquilityService {
+private final static String FIREHOSE_PATTERN = "druid:firehose:%s";
+
+private final static AllowableValue PT1M = new AllowableValue("PT1M", 
"1 minute", "1 minute");
+private final static AllowableValue PT10M = new 
AllowableValue("PT10M", "10 minutes", "10 minutes");
+private final static AllowableValue PT60M = new 
AllowableValue("PT60M", "60 minutes", "1 hour");
+
+private final static 

[jira] [Commented] (NIFI-4684) Add attribute prefix property to ConvertJSONToSQL

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296535#comment-16296535
 ] 

ASF subversion and git services commented on NIFI-4684:
---

Commit 612675e4283a22481ee76a0ad81a092f500150a9 in nifi's branch 
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=612675e ]

NIFI-4684 - Added SQL Parameter Attribute Prefix property to ConvertJSONToSQL

Signed-off-by: Pierre Villard 

This closes #2333.


> Add attribute prefix property to ConvertJSONToSQL
> -
>
> Key: NIFI-4684
> URL: https://issues.apache.org/jira/browse/NIFI-4684
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.5.0
>
>
> Some users have tried to use ConvertJSONToSQL to generate SQL to put into 
> Hive (possibly having to convert to HiveQL via ReplaceText). I'm not sure if 
> NIFI-4071 still applies, but even so, when generating/accepting prepared 
> statements, PutHiveQL expects attributes of the form 
> "hiveql.args.N.type/value", where ConvertJSONToSQL generates attributes of 
> the form "sql.args.N.type/value". 
> To accommodate both use cases, we could add a property to ConvertJSONToSQL 
> specifying the attribute prefix to use. It would default to "sql" to maintain 
> existing behavior but allow the user to change to something like "hiveql" if 
> they desire.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4684) Add attribute prefix property to ConvertJSONToSQL

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4684:
-
   Resolution: Fixed
Fix Version/s: 1.5.0
   Status: Resolved  (was: Patch Available)

> Add attribute prefix property to ConvertJSONToSQL
> -
>
> Key: NIFI-4684
> URL: https://issues.apache.org/jira/browse/NIFI-4684
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.5.0
>
>
> Some users have tried to use ConvertJSONToSQL to generate SQL to put into 
> Hive (possibly having to convert to HiveQL via ReplaceText). I'm not sure if 
> NIFI-4071 still applies, but even so, when generating/accepting prepared 
> statements, PutHiveQL expects attributes of the form 
> "hiveql.args.N.type/value", where ConvertJSONToSQL generates attributes of 
> the form "sql.args.N.type/value". 
> To accommodate both use cases, we could add a property to ConvertJSONToSQL 
> specifying the attribute prefix to use. It would default to "sql" to maintain 
> existing behavior but allow the user to change to something like "hiveql" if 
> they desire.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2333: NIFI-4684: Added SQL Parameter Attribute Prefix pro...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2333


---

[jira] [Commented] (NIFI-4684) Add attribute prefix property to ConvertJSONToSQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296536#comment-16296536
 ] 

ASF GitHub Bot commented on NIFI-4684:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2333


> Add attribute prefix property to ConvertJSONToSQL
> -
>
> Key: NIFI-4684
> URL: https://issues.apache.org/jira/browse/NIFI-4684
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.5.0
>
>
> Some users have tried to use ConvertJSONToSQL to generate SQL to put into 
> Hive (possibly having to convert to HiveQL via ReplaceText). I'm not sure if 
> NIFI-4071 still applies, but even so, when generating/accepting prepared 
> statements, PutHiveQL expects attributes of the form 
> "hiveql.args.N.type/value", where ConvertJSONToSQL generates attributes of 
> the form "sql.args.N.type/value". 
> To accommodate both use cases, we could add a property to ConvertJSONToSQL 
> specifying the attribute prefix to use. It would default to "sql" to maintain 
> existing behavior but allow the user to change to something like "hiveql" if 
> they desire.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4684) Add attribute prefix property to ConvertJSONToSQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296533#comment-16296533
 ] 

ASF GitHub Bot commented on NIFI-4684:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2333
  
+1, code LGTM and confirmed expected behavior when using this processor for 
Hive operations. Merging to master.


> Add attribute prefix property to ConvertJSONToSQL
> -
>
> Key: NIFI-4684
> URL: https://issues.apache.org/jira/browse/NIFI-4684
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Some users have tried to use ConvertJSONToSQL to generate SQL to put into 
> Hive (possibly having to convert to HiveQL via ReplaceText). I'm not sure if 
> NIFI-4071 still applies, but even so, when generating/accepting prepared 
> statements, PutHiveQL expects attributes of the form 
> "hiveql.args.N.type/value", where ConvertJSONToSQL generates attributes of 
> the form "sql.args.N.type/value". 
> To accommodate both use cases, we could add a property to ConvertJSONToSQL 
> specifying the attribute prefix to use. It would default to "sql" to maintain 
> existing behavior but allow the user to change to something like "hiveql" if 
> they desire.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2333: NIFI-4684: Added SQL Parameter Attribute Prefix property t...

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2333
  
+1, code LGTM and confirmed expected behavior when using this processor for 
Hive operations. Merging to master.


---

[jira] [Commented] (NIFI-4709) ListAzureBlobStorage misunderstands target system timestamp precision as Minutes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296526#comment-16296526
 ] 

ASF GitHub Bot commented on NIFI-4709:
--

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/2354

NIFI-4709: Fixed ListAzureBlobStorage timestamp precision handling.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-4709

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2354.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2354


commit d4ece742e111a8d0fc7292d5960c055c29fca215
Author: Koji Kawamura 
Date:   2017-12-19T09:25:23Z

NIFI-4709: Fixed ListAzureBlobStorage timestamp precision handling.




> ListAzureBlobStorage misunderstands target system timestamp precision as 
> Minutes
> 
>
> Key: NIFI-4709
> URL: https://issues.apache.org/jira/browse/NIFI-4709
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>
> NIFI-4069 added target system timestamp detection for List processors. 
> Defaults to auto detection. Most sub classes added 'Target System Timestamp 
> Precision' to their 'getSupportedPropertyDescriptors' method. But 
> ListAzureBlobStorage didn't.
> Even though 'Target System Timestamp Precision' property has a default value, 
> if it's not included in getSupportedPropertyDescriptors method, the property 
> value becomes null, instead of the default value. This combination is not 
> handled well in AbstractListProcessor currently. That makes 
> ListAzureBlobStorage behaves as if Azure Blob Storage time precision is in 
> Minutes while it actually has Seconds precision. Incurs longer time for blob 
> files to be picked than required.
> Not having 'Target System Timestamp Precision' at ListAzureBlobStorage seems 
> reasonable as the processor interact with only Azure Blob Storage, and its 
> timestamp precision should be fixed. AbstractListProcessor should provide an 
> extension point for sub-classes to define default precision. In case for 
> Azure Blob, it's SECONDS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4709) ListAzureBlobStorage misunderstands target system timestamp precision as Minutes

[Koji Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-4709:

Status: Patch Available  (was: Open)

> ListAzureBlobStorage misunderstands target system timestamp precision as 
> Minutes
> 
>
> Key: NIFI-4709
> URL: https://issues.apache.org/jira/browse/NIFI-4709
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>
> NIFI-4069 added target system timestamp detection for List processors. 
> Defaults to auto detection. Most sub classes added 'Target System Timestamp 
> Precision' to their 'getSupportedPropertyDescriptors' method. But 
> ListAzureBlobStorage didn't.
> Even though 'Target System Timestamp Precision' property has a default value, 
> if it's not included in getSupportedPropertyDescriptors method, the property 
> value becomes null, instead of the default value. This combination is not 
> handled well in AbstractListProcessor currently. That makes 
> ListAzureBlobStorage behaves as if Azure Blob Storage time precision is in 
> Minutes while it actually has Seconds precision. Incurs longer time for blob 
> files to be picked than required.
> Not having 'Target System Timestamp Precision' at ListAzureBlobStorage seems 
> reasonable as the processor interact with only Azure Blob Storage, and its 
> timestamp precision should be fixed. AbstractListProcessor should provide an 
> extension point for sub-classes to define default precision. In case for 
> Azure Blob, it's SECONDS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4696) Support concurrent tasks in PutHiveStreaming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296525#comment-16296525
 ] 

ASF GitHub Bot commented on NIFI-4696:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2342#discussion_r157701335
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/PutHiveStreaming.java
 ---
@@ -156,7 +158,9 @@
 .displayName("Hive Configuration Resources")
 .description("A file or comma separated list of files which 
contains the Hive configuration (hive-site.xml, e.g.). Without this, Hadoop "
 + "will search the classpath for a 'hive-site.xml' 
file or will revert to a default configuration. Note that to enable 
authentication "
-+ "with Kerberos e.g., the appropriate properties must 
be set in the configuration files. Please see the Hive documentation for more 
details.")
++ "with Kerberos e.g., the appropriate properties must 
be set in the configuration files. Also note that if Max Concurrent Tasks is 
set "
++ "to a number greater than one, the 
'hcatalog.hive.client.cache.disabled' property will be forced to 'true' to 
avoid concurrency issues. "
--- End diff --

I know you reference the Hive documentation but shouldn't we explicitly say 
that it's not possible to concurrently write in the same database.table?


> Support concurrent tasks in PutHiveStreaming
> 
>
> Key: NIFI-4696
> URL: https://issues.apache.org/jira/browse/NIFI-4696
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Currently PutHiveStreaming (PHS) can only support a single task at a time. 
> Before NIFI-4342, that meant each target table would need its own PHS 
> instance, which can be cumbersome with large numbers of tables. After 
> NIFI-4342, Expression Language could be used for SDLC purposes 
> (database/table changes between development and production, e.g.).
> However it would be nice to be able to support at least database/table names 
> using flow file attributes, and also to support multiple tasks to handle them 
> concurrently. Due to the nature of PHS and the Streaming Ingest APIs (and 
> implementation), it is likely not prudent to allow two tasks to write to the 
> same table and partition at the same time.
> I propose adding flow file attribute EL evaluation where prudent, and 
> allowing per-table concurrency in PHS. A thread will attempt to get a lock on 
> a table, and if it cannot, will rollback and return.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2354: NIFI-4709: Fixed ListAzureBlobStorage timestamp pre...

[ijokarumawak]

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/2354

NIFI-4709: Fixed ListAzureBlobStorage timestamp precision handling.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-4709

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2354.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2354


commit d4ece742e111a8d0fc7292d5960c055c29fca215
Author: Koji Kawamura 
Date:   2017-12-19T09:25:23Z

NIFI-4709: Fixed ListAzureBlobStorage timestamp precision handling.




---

[GitHub] nifi pull request #2342: NIFI-4696: Add Flowfile attribute EL support and pe...

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2342#discussion_r157701335
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/PutHiveStreaming.java
 ---
@@ -156,7 +158,9 @@
 .displayName("Hive Configuration Resources")
 .description("A file or comma separated list of files which 
contains the Hive configuration (hive-site.xml, e.g.). Without this, Hadoop "
 + "will search the classpath for a 'hive-site.xml' 
file or will revert to a default configuration. Note that to enable 
authentication "
-+ "with Kerberos e.g., the appropriate properties must 
be set in the configuration files. Please see the Hive documentation for more 
details.")
++ "with Kerberos e.g., the appropriate properties must 
be set in the configuration files. Also note that if Max Concurrent Tasks is 
set "
++ "to a number greater than one, the 
'hcatalog.hive.client.cache.disabled' property will be forced to 'true' to 
avoid concurrency issues. "
--- End diff --

I know you reference the Hive documentation but shouldn't we explicitly say 
that it's not possible to concurrently write in the same database.table?


---

[jira] [Updated] (NIFI-4689) Write Ahead Provenance Repository properties missing from nifi.properties file

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4689:
-
   Resolution: Fixed
Fix Version/s: 1.5.0
   Status: Resolved  (was: Patch Available)

> Write Ahead Provenance Repository properties missing from nifi.properties file
> --
>
> Key: NIFI-4689
> URL: https://issues.apache.org/jira/browse/NIFI-4689
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.4.0
> Environment: N/A
>Reporter: Matthew Clarke
>Assignee: Matt Gilman
>Priority: Minor
>  Labels: newbie
> Fix For: 1.5.0
>
>
> The following Write Ahead Provenance Repository properties are missing from 
> nifi.properties file:
> 1. nifi.provenance.repository.concurrent.merge.threads
> 2. nifi.provenance.repository.warm.cache.frequency
> These properties are documented in the NiFi admin Guide.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4689) Write Ahead Provenance Repository properties missing from nifi.properties file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296461#comment-16296461
 ] 

ASF GitHub Bot commented on NIFI-4689:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2352


> Write Ahead Provenance Repository properties missing from nifi.properties file
> --
>
> Key: NIFI-4689
> URL: https://issues.apache.org/jira/browse/NIFI-4689
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.4.0
> Environment: N/A
>Reporter: Matthew Clarke
>Assignee: Matt Gilman
>Priority: Minor
>  Labels: newbie
>
> The following Write Ahead Provenance Repository properties are missing from 
> nifi.properties file:
> 1. nifi.provenance.repository.concurrent.merge.threads
> 2. nifi.provenance.repository.warm.cache.frequency
> These properties are documented in the NiFi admin Guide.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2352: NIFI-4689: WriteAheadProvenance Properties

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2352


---

[jira] [Commented] (NIFI-4689) Write Ahead Provenance Repository properties missing from nifi.properties file

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296457#comment-16296457
 ] 

ASF subversion and git services commented on NIFI-4689:
---

Commit 17718940d7e5aaa2ec8682c6aedd03736ec4a0b6 in nifi's branch 
refs/heads/master from [~mcgilman]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=1771894 ]

NIFI-4689 - Ensuring all provenance properties are represented in 
nifi.properties.

Signed-off-by: Pierre Villard 

This closes #2352.


> Write Ahead Provenance Repository properties missing from nifi.properties file
> --
>
> Key: NIFI-4689
> URL: https://issues.apache.org/jira/browse/NIFI-4689
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.4.0
> Environment: N/A
>Reporter: Matthew Clarke
>Assignee: Matt Gilman
>Priority: Minor
>  Labels: newbie
>
> The following Write Ahead Provenance Repository properties are missing from 
> nifi.properties file:
> 1. nifi.provenance.repository.concurrent.merge.threads
> 2. nifi.provenance.repository.warm.cache.frequency
> These properties are documented in the NiFi admin Guide.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4689) Write Ahead Provenance Repository properties missing from nifi.properties file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296453#comment-16296453
 ] 

ASF GitHub Bot commented on NIFI-4689:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2352
  
Checked that properties are defined in the pom.xml of nifi-resources. 
Confirmed nifi.properties content after a build. +1, merging to master.


> Write Ahead Provenance Repository properties missing from nifi.properties file
> --
>
> Key: NIFI-4689
> URL: https://issues.apache.org/jira/browse/NIFI-4689
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.4.0
> Environment: N/A
>Reporter: Matthew Clarke
>Assignee: Matt Gilman
>Priority: Minor
>  Labels: newbie
>
> The following Write Ahead Provenance Repository properties are missing from 
> nifi.properties file:
> 1. nifi.provenance.repository.concurrent.merge.threads
> 2. nifi.provenance.repository.warm.cache.frequency
> These properties are documented in the NiFi admin Guide.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2352: NIFI-4689: WriteAheadProvenance Properties

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2352
  
Checked that properties are defined in the pom.xml of nifi-resources. 
Confirmed nifi.properties content after a build. +1, merging to master.


---