[jira] [Commented] (NIFI-5031) AvroSchemaRegistry not properly initialized on NiFi restart

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420085#comment-16420085
 ] 

ASF GitHub Bot commented on NIFI-5031:
--

Github user junegunn closed the pull request at:

https://github.com/apache/nifi/pull/2595


> AvroSchemaRegistry not properly initialized on NiFi restart
> ---
>
> Key: NIFI-5031
> URL: https://issues.apache.org/jira/browse/NIFI-5031
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Junegunn Choi
>Priority: Major
>
> When a NiFi instance is restarted, the internal \{{recordSchemas}} map of 
> AvroSchemaRegistry is not populated. The map is only updated when the content 
> of a dynamic property is updated, so unless we manually update all dynamic 
> properties, schema lookups will fail.
> We should rebuild the map from dynamic properties when the service is 
> enabled. Let me post a patch for this via GitHub pull request.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (NIFI-5031) AvroSchemaRegistry not properly initialized on NiFi restart

[Junegunn Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junegunn Choi resolved NIFI-5031.
-
Resolution: Duplicate

> AvroSchemaRegistry not properly initialized on NiFi restart
> ---
>
> Key: NIFI-5031
> URL: https://issues.apache.org/jira/browse/NIFI-5031
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Junegunn Choi
>Priority: Major
>
> When a NiFi instance is restarted, the internal \{{recordSchemas}} map of 
> AvroSchemaRegistry is not populated. The map is only updated when the content 
> of a dynamic property is updated, so unless we manually update all dynamic 
> properties, schema lookups will fail.
> We should rebuild the map from dynamic properties when the service is 
> enabled. Let me post a patch for this via GitHub pull request.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2595: NIFI-5031 Initialize AvroSchemaRegistry from dynami...

[junegunn]

Github user junegunn closed the pull request at:

https://github.com/apache/nifi/pull/2595


---

[jira] [Commented] (NIFI-5031) AvroSchemaRegistry not properly initialized on NiFi restart

[Junegunn Choi (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420084#comment-16420084
 ] 

Junegunn Choi commented on NIFI-5031:
-

{quote}maybe that is what you saw, but that is resolved now.
{quote}
Confirmed. I was using the master branch cloned last week and I can confirm 
that the problem is already fixed now. Thanks!

> AvroSchemaRegistry not properly initialized on NiFi restart
> ---
>
> Key: NIFI-5031
> URL: https://issues.apache.org/jira/browse/NIFI-5031
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Junegunn Choi
>Priority: Major
>
> When a NiFi instance is restarted, the internal \{{recordSchemas}} map of 
> AvroSchemaRegistry is not populated. The map is only updated when the content 
> of a dynamic property is updated, so unless we manually update all dynamic 
> properties, schema lookups will fail.
> We should rebuild the map from dynamic properties when the service is 
> enabled. Let me post a patch for this via GitHub pull request.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (NIFI-5033) Cannot update variable referenced in restricted components

[Matt Gilman (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419993#comment-16419993
 ] 

Matt Gilman edited comment on NIFI-5033 at 3/30/18 12:13 AM:
-

Hey [~pvillard] thanks for filing this! I see what's going on... There are 
actually two issues here one of which is a regression from 1.5.0.

1) The new granular restricted components introduced custom implementations of 
checkAuthorization (happens if we are checking a user's permissions so we can 
enable some UI feature) and authorize (happens if a user is actually attempting 
to interact with some component). By its signature, checkAuthorization should 
never throw any exceptions. If the user is unauthorized it should return 
AuthorizationResult.Denied. Conservely, authorize should throw an 
AccessDeniedException when the user is unauthorized. The regression in 1.6.0 is 
that we introduced a checkAuthorization method which throws an 
AccessDeniendException when the user is null. By its definition, it should 
simply return AuthorizationResult.Denied.

2) The other issue that was present in previous releases is that the user is 
null here. This should not be the case even in unsecured mode.  Instead, the 
user object should be populated with an anonymous user. The request in question 
is an asynchronous request meaning we hold a reference to the user in memory to 
process each step. By passing a null user object, it likely meant that an 
intermediate step of the variable update request returned the wrong permission 
indicators (by invoking checkAuthorization). In this case, it stated the user 
was not allowed to access some component since the user reference was not used 
when generated a response for the client. However, because it was an 
intermediate step and not the final result (which did correctly use the user 
reference) this was unnoticed.

 


was (Author: mcgilman):
Hey [~pvillard] thanks for filing this! I see what's going on... There are 
actually two issues here one of which is a regression from 1.5.0.

1) The new granular restricted components introduced custom implementations of 
checkAuthorization (happens if we are checking a user's permissions so we can 
enable some UI feature) and authorize (happens if a user is actually attempting 
to interact with some component). By its signature, checkAuthorization should 
never throw any exceptions. If the user is unauthorized it should return 
AuthorizationResult.Denied. Conservely, authorize should throw an 
AccessDeniedException when the user is unauthorized. The regression in 1.6.0 is 
that we introduced a checkAuthorization method which throws an 
AccessDeniendException when the user is null. By its definition, it should 
simply return AuthorizationResult.Denied.

2) The other issue that was present in previous releases is that the user is 
null here. This should not be the case even in unsecured mode.  Instead, the 
user object should be populated with an anonymous user. The request in question 
is an asynchronous request meaning we hold a reference to the user in memory to 
process each step. The passing a null user object, it likely meant that an 
intermediate step of the variable update request returned the wrong permission 
indicators (invoking checkAuthorization). In this case, it stated the user was 
not allowed to access some component since the user reference was not used when 
generated a response for the client. However, because it was an intermediate 
step and not the final result (which did correctly use the user reference) this 
was unnoticed.

 

> Cannot update variable referenced in restricted components
> --
>
> Key: NIFI-5033
> URL: https://issues.apache.org/jira/browse/NIFI-5033
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Pierre Villard
>Priority: Blocker
>
> When updating a variable at pg level that references a restricted component 
> it will fail. It seems the code is the same for secured and unsecured 
> instance and it fails when NiFi is unsecured since the user is unknown.
> It seems it has been introduced by NIFI-4885.
> {noformat}
> 2018-03-29 21:10:30,913 INFO [Variable Registry Update Thread] 
> o.a.nifi.web.api.ProcessGroupResource In order to update Variable Registry 
> for Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50, Stopping 
> Processors
> 2018-03-29 21:10:30,913 INFO [Variable Registry Update Thread] 
> o.a.nifi.web.api.ProcessGroupResource In order to update Variable Registry 
> for Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50, Disabling 
> Controller Services
> 2018-03-29 21:10:30,913 INFO [Variable Registry Update Thread] 
> o.a.nifi.web.api.ProcessGroupResource In order to update Variable 

[jira] [Commented] (NIFI-5033) Cannot update variable referenced in restricted components

[Matt Gilman (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419993#comment-16419993
 ] 

Matt Gilman commented on NIFI-5033:
---

Hey [~pvillard] thanks for filing this! I see what's going on... There are 
actually two issues here one of which is a regression from 1.5.0.

1) The new granular restricted components introduced custom implementations of 
checkAuthorization (happens if we are checking a user's permissions so we can 
enable some UI feature) and authorize (happens if a user is actually attempting 
to interact with some component). By its signature, checkAuthorization should 
never throw any exceptions. If the user is unauthorized it should return 
AuthorizationResult.Denied. Conservely, authorize should throw an 
AccessDeniedException when the user is unauthorized. The regression in 1.6.0 is 
that we introduced a checkAuthorization method which throws an 
AccessDeniendException when the user is null. By its definition, it should 
simply return AuthorizationResult.Denied.

2) The other issue that was present in previous releases is that the user is 
null here. This should not be the case even in unsecured mode.  Instead, the 
user object should be populated with an anonymous user. The request in question 
is an asynchronous request meaning we hold a reference to the user in memory to 
process each step. The passing a null user object, it likely meant that an 
intermediate step of the variable update request returned the wrong permission 
indicators (invoking checkAuthorization). In this case, it stated the user was 
not allowed to access some component since the user reference was not used when 
generated a response for the client. However, because it was an intermediate 
step and not the final result (which did correctly use the user reference) this 
was unnoticed.

 

> Cannot update variable referenced in restricted components
> --
>
> Key: NIFI-5033
> URL: https://issues.apache.org/jira/browse/NIFI-5033
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Pierre Villard
>Priority: Blocker
>
> When updating a variable at pg level that references a restricted component 
> it will fail. It seems the code is the same for secured and unsecured 
> instance and it fails when NiFi is unsecured since the user is unknown.
> It seems it has been introduced by NIFI-4885.
> {noformat}
> 2018-03-29 21:10:30,913 INFO [Variable Registry Update Thread] 
> o.a.nifi.web.api.ProcessGroupResource In order to update Variable Registry 
> for Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50, Stopping 
> Processors
> 2018-03-29 21:10:30,913 INFO [Variable Registry Update Thread] 
> o.a.nifi.web.api.ProcessGroupResource In order to update Variable Registry 
> for Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50, Disabling 
> Controller Services
> 2018-03-29 21:10:30,913 INFO [Variable Registry Update Thread] 
> o.a.nifi.web.api.ProcessGroupResource In order to update Variable Registry 
> for Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50, Applying 
> updates to Variable Registry
> 2018-03-29 21:10:30,915 ERROR [Variable Registry Update Thread] 
> o.a.nifi.web.api.ProcessGroupResource Failed to update variable registry for 
> Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50
> org.apache.nifi.authorization.AccessDeniedException: Unknown user.
> at 
> org.apache.nifi.authorization.resource.RestrictedComponentsAuthorizableFactory$2.checkAuthorization(RestrictedComponentsAuthorizableFactory.java:68)
> at 
> org.apache.nifi.controller.ConfiguredComponent.checkAuthorization(ConfiguredComponent.java:129)
> at 
> org.apache.nifi.authorization.resource.Authorizable.checkAuthorization(Authorizable.java:183)
> at 
> org.apache.nifi.authorization.resource.Authorizable.isAuthorized(Authorizable.java:70)
> at 
> org.apache.nifi.web.api.dto.DtoFactory.createPermissionsDto(DtoFactory.java:1798)
> at 
> org.apache.nifi.web.api.dto.DtoFactory.createPermissionsDto(DtoFactory.java:1785)
> at 
> org.apache.nifi.web.api.dto.DtoFactory.lambda$createAffectedComponentEntities$73(DtoFactory.java:2485)
> at 
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at 
> java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1540)
> at 
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at 
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at 
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at 
> 

[jira] [Comment Edited] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419722#comment-16419722
 ] 

Otto Fowler edited comment on NIFI-3576 at 3/29/18 9:43 PM:


{code:java}
if ( (hits.size() == 0 && queryInfoRouteStrategy == 
QueryInfoRouteStrategy.NOHIT)
|| queryInfoRouteStrategy == QueryInfoRouteStrategy.ALWAYS) {
FlowFile queryInfo = flowFile == null ? session.create() : 
session.create(flowFile);
session.putAttribute(queryInfo, "es.query.url", url.toExternalForm());
session.putAttribute(queryInfo, "es.query.hitCount", 
String.valueOf(hits.size()));
session.putAttribute(queryInfo, MIME_TYPE.key(), "application/json");
  
session.transfer(queryInfo,REL_QUERY_INFO);
}
{code}
like that


was (Author: ottobackwards):
{code:java}
if ( (hits.size() == 0 && queryInfoRouteStrategy == 
QueryInfoRouteStrategy.NOHIT)
|| queryInfoRouteStrategy == QueryInfoRouteStrategy.ALWAYS) {
FlowFile queryInfo = flowFile == null ? session.create() : 
session.create(flowFile);
session.putAttribute(queryInfo, "es.query.url", url.toExternalForm());
session.putAttribute(queryInfo, "es.query.hitCount", 
String.valueOf(hits.size()));
session.putAttribute(queryInfo, MIME_TYPE.key(), "application/json");
session.importFrom(new ByteArrayInputStream(bodyBytes),queryInfo);
session.transfer(queryInfo,REL_QUERY_INFO);
}
{code}
like that

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5033) Cannot update variable referenced in restricted components

[Pierre Villard (JIRA)]

Pierre Villard created NIFI-5033:


 Summary: Cannot update variable referenced in restricted components
 Key: NIFI-5033
 URL: https://issues.apache.org/jira/browse/NIFI-5033
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.6.0
Reporter: Pierre Villard


When updating a variable at pg level that references a restricted component it 
will fail. It seems the code is the same for secured and unsecured instance and 
it fails when NiFi is unsecured since the user is unknown.

It seems it has been introduced by NIFI-4885.
{noformat}
2018-03-29 21:10:30,913 INFO [Variable Registry Update Thread] 
o.a.nifi.web.api.ProcessGroupResource In order to update Variable Registry for 
Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50, Stopping Processors
2018-03-29 21:10:30,913 INFO [Variable Registry Update Thread] 
o.a.nifi.web.api.ProcessGroupResource In order to update Variable Registry for 
Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50, Disabling 
Controller Services
2018-03-29 21:10:30,913 INFO [Variable Registry Update Thread] 
o.a.nifi.web.api.ProcessGroupResource In order to update Variable Registry for 
Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50, Applying updates to 
Variable Registry
2018-03-29 21:10:30,915 ERROR [Variable Registry Update Thread] 
o.a.nifi.web.api.ProcessGroupResource Failed to update variable registry for 
Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50
org.apache.nifi.authorization.AccessDeniedException: Unknown user.
at 
org.apache.nifi.authorization.resource.RestrictedComponentsAuthorizableFactory$2.checkAuthorization(RestrictedComponentsAuthorizableFactory.java:68)
at 
org.apache.nifi.controller.ConfiguredComponent.checkAuthorization(ConfiguredComponent.java:129)
at 
org.apache.nifi.authorization.resource.Authorizable.checkAuthorization(Authorizable.java:183)
at 
org.apache.nifi.authorization.resource.Authorizable.isAuthorized(Authorizable.java:70)
at 
org.apache.nifi.web.api.dto.DtoFactory.createPermissionsDto(DtoFactory.java:1798)
at 
org.apache.nifi.web.api.dto.DtoFactory.createPermissionsDto(DtoFactory.java:1785)
at 
org.apache.nifi.web.api.dto.DtoFactory.lambda$createAffectedComponentEntities$73(DtoFactory.java:2485)
at 
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1540)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at 
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at 
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at 
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at 
org.apache.nifi.web.api.dto.DtoFactory.createAffectedComponentEntities(DtoFactory.java:2489)
at 
org.apache.nifi.web.api.dto.DtoFactory.createVariableRegistryDto(DtoFactory.java:2507)
at 
org.apache.nifi.web.StandardNiFiServiceFacade.lambda$updateVariableRegistry$36(StandardNiFiServiceFacade.java:950)
at 
org.apache.nifi.web.StandardNiFiServiceFacade$1.update(StandardNiFiServiceFacade.java:721)
at 
org.apache.nifi.web.revision.NaiveRevisionManager.updateRevision(NaiveRevisionManager.java:120)
at 
org.apache.nifi.web.StandardNiFiServiceFacade.updateComponent(StandardNiFiServiceFacade.java:712)
at 
org.apache.nifi.web.StandardNiFiServiceFacade.updateVariableRegistry(StandardNiFiServiceFacade.java:947)
at 
org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
at 
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at 
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at 
org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithWriteLock(NiFiServiceFacadeLock.java:173)
at 
org.apache.nifi.web.NiFiServiceFacadeLock.updateLock(NiFiServiceFacadeLock.java:66)
at sun.reflect.GeneratedMethodAccessor585.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at 

[jira] [Commented] (NIFI-4932) Enable S2S work behind a Reverse Proxy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419744#comment-16419744
 ] 

ASF GitHub Bot commented on NIFI-4932:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2510
  
@ijokarumawak Just wanted to add that I have verified this capability 
running standalone and clustered and everything seems to be working nicely. 
Just a couple more minor error handling cases. Thanks!


> Enable S2S work behind a Reverse Proxy
> --
>
> Key: NIFI-4932
> URL: https://issues.apache.org/jira/browse/NIFI-4932
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
>
> Currently, NiFi UI and REST API work through a reverse proxy, but NiFi 
> Site-to-Site does not. The core issue is how a NiFi node introduce remote 
> peers to Site-to-Site clients. NiFi should provide more flexible 
> configuration so that user can define remote Site-to-Site endpoints those can 
> work for both routes, through a reverse proxy, and directly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2510: NIFI-4932: Enable S2S work behind a Reverse Proxy

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2510
  
@ijokarumawak Just wanted to add that I have verified this capability 
running standalone and clustered and everything seems to be working nicely. 
Just a couple more minor error handling cases. Thanks!


---

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419736#comment-16419736
 ] 

Otto Fowler commented on NIFI-3576:
---

What i'm doing:
 * new optional route with property to enable
 * property sets strategy for sending query info to ALWAYS, NEVER or NOHIT
 * another property to include the query URL in the Success FF

 

 

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4932) Enable S2S work behind a Reverse Proxy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419728#comment-16419728
 ] 

ASF GitHub Bot commented on NIFI-4932:
--

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2510#discussion_r178175082
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/PeerDescriptionModifier.java
 ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.remote;
+
+import org.apache.nifi.attribute.expression.language.PreparedQuery;
+import org.apache.nifi.attribute.expression.language.Query;
+import org.apache.nifi.remote.protocol.SiteToSiteTransportProtocol;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static java.lang.String.format;
+import static org.apache.commons.lang3.StringUtils.isBlank;
+
+public class PeerDescriptionModifier {
+
+private static final Logger logger = 
LoggerFactory.getLogger(PeerDescriptionModifier.class);
+
+public enum RequestType {
+SiteToSiteDetail,
+Peers
+}
+
+private static class Route {
+private String name;
+private SiteToSiteTransportProtocol protocol;
+private PreparedQuery predicate;
+private PreparedQuery hostname;
+private PreparedQuery port;
+private PreparedQuery secure;
+
+private Route validate() {
+if (hostname == null) {
+throw new IllegalArgumentException(
+format("Found an invalid Site-to-Site route 
definition [%s] 'hostname' is not specified.", name));
+}
+if (port == null) {
+throw new IllegalArgumentException(
+format("Found an invalid Site-to-Site route 
definition [%s] 'port' is not specified.", name));
+}
+return this;
+}
+
+private PeerDescription getTarget(final Map 
variables) {
+final String targetHostName = 
hostname.evaluateExpressions(variables, null);
+if (isBlank(targetHostName)) {
+throw new IllegalStateException("Target hostname was not 
resolved for the route definition " + name);
+}
+
+final String targetPortStr = 
port.evaluateExpressions(variables, null);
+if (isBlank(targetPortStr)) {
+throw new IllegalStateException("Target port was not 
resolved for the route definition " + name);
+}
+
+final String targetIsSecure = secure == null ? null : 
secure.evaluateExpressions(variables, null);
+return new PeerDescription(targetHostName, 
Integer.valueOf(targetPortStr), Boolean.valueOf(targetIsSecure));
+}
+}
+
+private Map routes;
+
+
+private static final String PROPERTY_PREFIX = "nifi.remote.route.";
+
+public PeerDescriptionModifier(final NiFiProperties properties) {
+final Map routeDefinitions = 
properties.getPropertyKeys().stream()
+.filter(propertyKey -> 
propertyKey.startsWith(PROPERTY_PREFIX))
+.collect(Collectors.groupingBy(propertyKey -> 
propertyKey.substring(PROPERTY_PREFIX.length(), propertyKey.lastIndexOf('.';
+
+routes = routeDefinitions.entrySet().stream().map(routeDefinition 
-> {
+final Route route = new Route();
+// E.g. raw.example1, http.example2
+final String[] protocolAndRoutingName = 
routeDefinition.getKey().split("\\.");
+route.protocol = 

[jira] [Comment Edited] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419722#comment-16419722
 ] 

Otto Fowler edited comment on NIFI-3576 at 3/29/18 8:38 PM:


{code:java}
if ( (hits.size() == 0 && queryInfoRouteStrategy == 
QueryInfoRouteStrategy.NOHIT)
|| queryInfoRouteStrategy == QueryInfoRouteStrategy.ALWAYS) {
FlowFile queryInfo = flowFile == null ? session.create() : 
session.create(flowFile);
session.putAttribute(queryInfo, "es.query.url", url.toExternalForm());
session.putAttribute(queryInfo, "es.query.hitCount", 
String.valueOf(hits.size()));
session.putAttribute(queryInfo, MIME_TYPE.key(), "application/json");
session.importFrom(new ByteArrayInputStream(bodyBytes),queryInfo);
session.transfer(queryInfo,REL_QUERY_INFO);
}
{code}
like that


was (Author: ottobackwards):
{code:java}
if ( (hits.size() == 0 && queryInfoRouteStrategy == 
QueryInfoRouteStrategy.NOHIT)
|| queryInfoRouteStrategy == QueryInfoRouteStrategy.ALWAYS) {
FlowFile queryInfo = flowFile == null ? session.create() : 
session.create(flowFile);
session.putAttribute(queryInfo, "es.query.url", url.toExternalForm());
session.putAttribute(queryInfo, "es.query.hitCount", 
String.valueOf(hits.size()));
session.putAttribute(queryInfo, MIME_TYPE.key(), "application/json");
session.importFrom(new ByteArrayInputStream(bodyBytes),queryInfo);
session.transfer(queryInfo,REL_QUERY_INFO);
}
{code}
like that

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2510: NIFI-4932: Enable S2S work behind a Reverse Proxy

[mcgilman]

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2510#discussion_r178175082
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/PeerDescriptionModifier.java
 ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.remote;
+
+import org.apache.nifi.attribute.expression.language.PreparedQuery;
+import org.apache.nifi.attribute.expression.language.Query;
+import org.apache.nifi.remote.protocol.SiteToSiteTransportProtocol;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static java.lang.String.format;
+import static org.apache.commons.lang3.StringUtils.isBlank;
+
+public class PeerDescriptionModifier {
+
+private static final Logger logger = 
LoggerFactory.getLogger(PeerDescriptionModifier.class);
+
+public enum RequestType {
+SiteToSiteDetail,
+Peers
+}
+
+private static class Route {
+private String name;
+private SiteToSiteTransportProtocol protocol;
+private PreparedQuery predicate;
+private PreparedQuery hostname;
+private PreparedQuery port;
+private PreparedQuery secure;
+
+private Route validate() {
+if (hostname == null) {
+throw new IllegalArgumentException(
+format("Found an invalid Site-to-Site route 
definition [%s] 'hostname' is not specified.", name));
+}
+if (port == null) {
+throw new IllegalArgumentException(
+format("Found an invalid Site-to-Site route 
definition [%s] 'port' is not specified.", name));
+}
+return this;
+}
+
+private PeerDescription getTarget(final Map 
variables) {
+final String targetHostName = 
hostname.evaluateExpressions(variables, null);
+if (isBlank(targetHostName)) {
+throw new IllegalStateException("Target hostname was not 
resolved for the route definition " + name);
+}
+
+final String targetPortStr = 
port.evaluateExpressions(variables, null);
+if (isBlank(targetPortStr)) {
+throw new IllegalStateException("Target port was not 
resolved for the route definition " + name);
+}
+
+final String targetIsSecure = secure == null ? null : 
secure.evaluateExpressions(variables, null);
+return new PeerDescription(targetHostName, 
Integer.valueOf(targetPortStr), Boolean.valueOf(targetIsSecure));
+}
+}
+
+private Map routes;
+
+
+private static final String PROPERTY_PREFIX = "nifi.remote.route.";
+
+public PeerDescriptionModifier(final NiFiProperties properties) {
+final Map routeDefinitions = 
properties.getPropertyKeys().stream()
+.filter(propertyKey -> 
propertyKey.startsWith(PROPERTY_PREFIX))
+.collect(Collectors.groupingBy(propertyKey -> 
propertyKey.substring(PROPERTY_PREFIX.length(), propertyKey.lastIndexOf('.';
+
+routes = routeDefinitions.entrySet().stream().map(routeDefinition 
-> {
+final Route route = new Route();
+// E.g. raw.example1, http.example2
+final String[] protocolAndRoutingName = 
routeDefinition.getKey().split("\\.");
+route.protocol = 
SiteToSiteTransportProtocol.valueOf(protocolAndRoutingName[0].toUpperCase());
+route.name = protocolAndRoutingName[1];
+routeDefinition.getValue().forEach(propertyKey -> {
+final 

[jira] [Comment Edited] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419722#comment-16419722
 ] 

Otto Fowler edited comment on NIFI-3576 at 3/29/18 8:38 PM:


{code:java}
if ( (hits.size() == 0 && queryInfoRouteStrategy == 
QueryInfoRouteStrategy.NOHIT)
|| queryInfoRouteStrategy == QueryInfoRouteStrategy.ALWAYS) {
FlowFile queryInfo = flowFile == null ? session.create() : 
session.create(flowFile);
session.putAttribute(queryInfo, "es.query.url", url.toExternalForm());
session.putAttribute(queryInfo, "es.query.hitCount", 
String.valueOf(hits.size()));
session.putAttribute(queryInfo, MIME_TYPE.key(), "application/json");
session.importFrom(new ByteArrayInputStream(bodyBytes),queryInfo);
session.transfer(queryInfo,REL_QUERY_INFO);
}
{code}
like that


was (Author: ottobackwards):
{code:java}
if ( (hits.size() == 0 && queryInfoRouteStrategy == 
QueryInfoRouteStrategy.NOHIT)
|| queryInfoRouteStrategy == QueryInfoRouteStrategy.ALWAYS) {
FlowFile queryInfo = flowFile == null ? session.create() : 
session.create(flowFile);
session.putAttribute(queryInfo, "es.query.url", url.toExternalForm());
session.putAttribute(queryInfo, "es.query.hitCount", 
String.valueOf(hits.size()));
session.putAttribute(queryInfo, MIME_TYPE.key(), "application/json");
session.importFrom(new ByteArrayInputStream(bodyBytes),queryInfo);
}
{code}
like that

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4932) Enable S2S work behind a Reverse Proxy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419721#comment-16419721
 ] 

ASF GitHub Bot commented on NIFI-4932:
--

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2510#discussion_r178174688
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/PeerDescriptionModifier.java
 ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.remote;
+
+import org.apache.nifi.attribute.expression.language.PreparedQuery;
+import org.apache.nifi.attribute.expression.language.Query;
+import org.apache.nifi.remote.protocol.SiteToSiteTransportProtocol;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static java.lang.String.format;
+import static org.apache.commons.lang3.StringUtils.isBlank;
+
+public class PeerDescriptionModifier {
+
+private static final Logger logger = 
LoggerFactory.getLogger(PeerDescriptionModifier.class);
+
+public enum RequestType {
+SiteToSiteDetail,
+Peers
+}
+
+private static class Route {
+private String name;
+private SiteToSiteTransportProtocol protocol;
+private PreparedQuery predicate;
+private PreparedQuery hostname;
+private PreparedQuery port;
+private PreparedQuery secure;
+
+private Route validate() {
+if (hostname == null) {
+throw new IllegalArgumentException(
+format("Found an invalid Site-to-Site route 
definition [%s] 'hostname' is not specified.", name));
+}
+if (port == null) {
+throw new IllegalArgumentException(
+format("Found an invalid Site-to-Site route 
definition [%s] 'port' is not specified.", name));
+}
+return this;
+}
+
+private PeerDescription getTarget(final Map 
variables) {
+final String targetHostName = 
hostname.evaluateExpressions(variables, null);
+if (isBlank(targetHostName)) {
+throw new IllegalStateException("Target hostname was not 
resolved for the route definition " + name);
+}
+
+final String targetPortStr = 
port.evaluateExpressions(variables, null);
+if (isBlank(targetPortStr)) {
+throw new IllegalStateException("Target port was not 
resolved for the route definition " + name);
+}
+
+final String targetIsSecure = secure == null ? null : 
secure.evaluateExpressions(variables, null);
+return new PeerDescription(targetHostName, 
Integer.valueOf(targetPortStr), Boolean.valueOf(targetIsSecure));
+}
+}
+
+private Map routes;
+
+
+private static final String PROPERTY_PREFIX = "nifi.remote.route.";
+
+public PeerDescriptionModifier(final NiFiProperties properties) {
+final Map routeDefinitions = 
properties.getPropertyKeys().stream()
+.filter(propertyKey -> 
propertyKey.startsWith(PROPERTY_PREFIX))
+.collect(Collectors.groupingBy(propertyKey -> 
propertyKey.substring(PROPERTY_PREFIX.length(), propertyKey.lastIndexOf('.';
+
+routes = routeDefinitions.entrySet().stream().map(routeDefinition 
-> {
+final Route route = new Route();
+// E.g. raw.example1, http.example2
+final String[] protocolAndRoutingName = 
routeDefinition.getKey().split("\\.");
+route.protocol = 

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419722#comment-16419722
 ] 

Otto Fowler commented on NIFI-3576:
---

{code:java}
if ( (hits.size() == 0 && queryInfoRouteStrategy == 
QueryInfoRouteStrategy.NOHIT)
|| queryInfoRouteStrategy == QueryInfoRouteStrategy.ALWAYS) {
FlowFile queryInfo = flowFile == null ? session.create() : 
session.create(flowFile);
session.putAttribute(queryInfo, "es.query.url", url.toExternalForm());
session.putAttribute(queryInfo, "es.query.hitCount", 
String.valueOf(hits.size()));
session.putAttribute(queryInfo, MIME_TYPE.key(), "application/json");
session.importFrom(new ByteArrayInputStream(bodyBytes),queryInfo);
}
{code}
like that

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2510: NIFI-4932: Enable S2S work behind a Reverse Proxy

[mcgilman]

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2510#discussion_r178174688
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/PeerDescriptionModifier.java
 ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.remote;
+
+import org.apache.nifi.attribute.expression.language.PreparedQuery;
+import org.apache.nifi.attribute.expression.language.Query;
+import org.apache.nifi.remote.protocol.SiteToSiteTransportProtocol;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static java.lang.String.format;
+import static org.apache.commons.lang3.StringUtils.isBlank;
+
+public class PeerDescriptionModifier {
+
+private static final Logger logger = 
LoggerFactory.getLogger(PeerDescriptionModifier.class);
+
+public enum RequestType {
+SiteToSiteDetail,
+Peers
+}
+
+private static class Route {
+private String name;
+private SiteToSiteTransportProtocol protocol;
+private PreparedQuery predicate;
+private PreparedQuery hostname;
+private PreparedQuery port;
+private PreparedQuery secure;
+
+private Route validate() {
+if (hostname == null) {
+throw new IllegalArgumentException(
+format("Found an invalid Site-to-Site route 
definition [%s] 'hostname' is not specified.", name));
+}
+if (port == null) {
+throw new IllegalArgumentException(
+format("Found an invalid Site-to-Site route 
definition [%s] 'port' is not specified.", name));
+}
+return this;
+}
+
+private PeerDescription getTarget(final Map 
variables) {
+final String targetHostName = 
hostname.evaluateExpressions(variables, null);
+if (isBlank(targetHostName)) {
+throw new IllegalStateException("Target hostname was not 
resolved for the route definition " + name);
+}
+
+final String targetPortStr = 
port.evaluateExpressions(variables, null);
+if (isBlank(targetPortStr)) {
+throw new IllegalStateException("Target port was not 
resolved for the route definition " + name);
+}
+
+final String targetIsSecure = secure == null ? null : 
secure.evaluateExpressions(variables, null);
+return new PeerDescription(targetHostName, 
Integer.valueOf(targetPortStr), Boolean.valueOf(targetIsSecure));
+}
+}
+
+private Map routes;
+
+
+private static final String PROPERTY_PREFIX = "nifi.remote.route.";
+
+public PeerDescriptionModifier(final NiFiProperties properties) {
+final Map routeDefinitions = 
properties.getPropertyKeys().stream()
+.filter(propertyKey -> 
propertyKey.startsWith(PROPERTY_PREFIX))
+.collect(Collectors.groupingBy(propertyKey -> 
propertyKey.substring(PROPERTY_PREFIX.length(), propertyKey.lastIndexOf('.';
+
+routes = routeDefinitions.entrySet().stream().map(routeDefinition 
-> {
+final Route route = new Route();
+// E.g. raw.example1, http.example2
+final String[] protocolAndRoutingName = 
routeDefinition.getKey().split("\\.");
+route.protocol = 
SiteToSiteTransportProtocol.valueOf(protocolAndRoutingName[0].toUpperCase());
+route.name = protocolAndRoutingName[1];
+routeDefinition.getValue().forEach(propertyKey -> {
+final 

[jira] [Commented] (NIFI-5017) ConvertExcelToCSVProcessor should Support Expression Language for Number of Rows to Skip and Columns To Skip

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419718#comment-16419718
 ] 

ASF GitHub Bot commented on NIFI-5017:
--

Github user ak-arun commented on the issue:

https://github.com/apache/nifi/pull/2597
  
@pvillard31 : incorporated the recommended changes.  


> ConvertExcelToCSVProcessor  should Support Expression Language for Number of 
> Rows to Skip and Columns To Skip
> -
>
> Key: NIFI-5017
> URL: https://issues.apache.org/jira/browse/NIFI-5017
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Arun A K
>Priority: Minor
>
> The ConvertExcelToCSVProcessor should support expression language for fields 
> "Number of Rows to Skip" and "Columns To Skip". These two could be a 
> precomputed attribute or looked up value and need to be configurable via EL. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2597: NIFI-5017 expression language support for rows and columns...

[ak-arun]

Github user ak-arun commented on the issue:

https://github.com/apache/nifi/pull/2597
  
@pvillard31 : incorporated the recommended changes.  


---

[GitHub] nifi-minifi pull request #121: MINIFI-449 Support for converting VersionedFl...

[bbende]

GitHub user bbende opened a pull request:

https://github.com/apache/nifi-minifi/pull/121

MINIFI-449 Support for converting VersionedFlowSnapshot to YAML via m…

…inifi-toolkit

- Upgrading Jackson databind to 2.9.1 for consistency with other Jackson 
modules

Thank you for submitting a contribution to Apache NiFi - MiNiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi-minifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under minifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under minifi-assembly?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbende/nifi-minifi MINIFI-449-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi/pull/121.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #121


commit 3fbd117c1f359d7423a75f6f9280e2f86a3d84e8
Author: Bryan Bende 
Date:   2018-03-28T17:24:58Z

MINIFI-449 Support for converting VersionedFlowSnapshot to YAML via 
minifi-toolkit
- Upgrading Jackson databind to 2.9.1 for consistency with other Jackson 
modules




---

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419656#comment-16419656
 ] 

Joseph Percivall commented on NIFI-3576:


[~ottobackwards] I agree

 

[~panchicore] and [~wietze], would this path forward solve your use-cases?

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419653#comment-16419653
 ] 

Otto Fowler commented on NIFI-3576:
---

OK, this feels like something to start with.

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419639#comment-16419639
 ] 

Otto Fowler commented on NIFI-3576:
---

I like : never, no-hits, always!

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419637#comment-16419637
 ] 

Otto Fowler commented on NIFI-3576:
---

that and it would have the query as the FF content

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419631#comment-16419631
 ] 

Joseph Percivall commented on NIFI-3576:


Hmm, the attributes on a no-hits REL FF could be the same as a query-info REL 
FF. So the main differences would be the name and when it emits a FF. What if 
we took a best of both worlds and added the query-info relationship and for the 
property, it is an option of when to emit the FF to that relationship? The 
options would either be "never", "when there are no hits", or "always" (default 
to never).

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419621#comment-16419621
 ] 

Otto Fowler commented on NIFI-3576:
---

Still conforming to your above concept

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419620#comment-16419620
 ] 

Otto Fowler commented on NIFI-3576:
---

So - here is a question.

Would a no-hits be more useful than a query-info which had the query and the 
hitcount and other meta that we think of?

You are going to check attrs either way, and the query is good for debug

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419599#comment-16419599
 ] 

Joseph Percivall commented on NIFI-3576:


There are a couple examples, such as RouteOnAttribute, RouteOnContent, and 
QueryRecord but the best example for this use case is probably LookUpRecord[1].

Ah looks like it's still an open ticket to add DisplayName to Relationships[2]. 
Definitely, would rather not break people's flows. Let's just change up the 
description to better match.

 

[1] 
[https://github.com/apache/nifi/blob/d7347a2dc3f73e76d03b9b3ef95015dd539e16ac/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupRecord.java#L237]

[2] https://issues.apache.org/jira/browse/NIFI-3591

 

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419586#comment-16419586
 ] 

Otto Fowler commented on NIFI-3576:
---

I also, don't see a displayName() for RELs like there is for properties, should 
I worry about breaking people calling getRelationship(String name)?

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419569#comment-16419569
 ] 

Otto Fowler commented on NIFI-3576:
---

I am not sure how you make a relationship optional based on a property.  Is 
there an example?

I like the success to hits, that de-conflates things.

 

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419555#comment-16419555
 ] 

Joseph Percivall commented on NIFI-3576:


Ah, I see your point about the FlowFile per hit and agree that there's some 
weirdness with the attributes for an empty "hit" FlowFile. The actual "hit per 
FF" functionality kinda goes against what is documented so we should change 
that a bit. With that, I also agree that 3 isn't what we want.

 

So what if we changed the displayName for "success" to "hits" and add a new 
property to say whether to emit a FF to a 'no hits' relationship. The property 
would default to not emitting and the relationship would only be available when 
that prop is true. That way the functionality is fully backwards compatible.

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419548#comment-16419548
 ] 

Otto Fowler commented on NIFI-3576:
---

I'm sorry, I meant 3 is valid as first step, and instead of 1-2 the new 
REL_QUERY_INFO.

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419537#comment-16419537
 ] 

Otto Fowler commented on NIFI-3576:
---

I agree that it is valid to return no hits, if this were returning all hits as 
one OR one flow file per hit you can imagine just outputting "\{"hits":[]}.  
Since this outputs a file per hit, what the attributes would be on the 'empty' 
FF is where I'm left wondering.  I would think that doing   hit.number, 
nit.total  ( so 1 of 20 ) etc, like segment numbers would work ( and make sense 
if you did get hits as well).

 

I think 1 is a valid step.

 

Instead of 2 or 3, I was thinking it would be better to have a REL_QUERY_INFO, 
that had the query as the document, and the hits etc as attributes.

 

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419524#comment-16419524
 ] 

Joseph Percivall commented on NIFI-3576:


So to summarize we have a couple of different options:
 # A new 'original' relationship to route the original FlowFile which has an 
attribute like 'es.hits.total' - precedent set by InvokeHttp
 # A 'none found' relationship which routes a FlowFile when there are no hits - 
precedent set by FetchElasticsearch(5)
 # Emit a FlowFile to "Success" for but add an attribute 'es.hits.total' - 
matches the 'Success' relationship documentation which states "All FlowFiles 
that are read from Elasticsearch are routed to this relationship."

I'm leaning more towards option 3. It seems more in line with the original 
intent of the relationship, and the processor as a whole. It's querying for 
potential hits, not fetching a single document, so having no results but, a 
successful query is valid. Also, this means that users won't have an invalid 
processor on migration (the new relationships wouldn't be routed anywhere). 
Then we also aren't duplicating logic which could just be in a follow-up 
RouteOnAttribute processor.

Thoughts?

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5017) ConvertExcelToCSVProcessor should Support Expression Language for Number of Rows to Skip and Columns To Skip

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419513#comment-16419513
 ] 

ASF GitHub Bot commented on NIFI-5017:
--

Github user ak-arun commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178138228
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessorTest.java
 ---
@@ -187,6 +213,34 @@ public void testSkipColumns() throws Exception {
 "9.8765E+08,\n");
 }
 
+@Test
+public void testSkipColumnsWithEL() throws Exception {
+testRunner.enqueue(new 
File("src/test/resources/dataformatting.xlsx").toPath());
--- End diff --

@pvillard31 : okay got it. Will make this change. 


> ConvertExcelToCSVProcessor  should Support Expression Language for Number of 
> Rows to Skip and Columns To Skip
> -
>
> Key: NIFI-5017
> URL: https://issues.apache.org/jira/browse/NIFI-5017
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Arun A K
>Priority: Minor
>
> The ConvertExcelToCSVProcessor should support expression language for fields 
> "Number of Rows to Skip" and "Columns To Skip". These two could be a 
> precomputed attribute or looked up value and need to be configurable via EL. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2597: NIFI-5017 expression language support for rows and ...

[ak-arun]

Github user ak-arun commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178138228
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessorTest.java
 ---
@@ -187,6 +213,34 @@ public void testSkipColumns() throws Exception {
 "9.8765E+08,\n");
 }
 
+@Test
+public void testSkipColumnsWithEL() throws Exception {
+testRunner.enqueue(new 
File("src/test/resources/dataformatting.xlsx").toPath());
--- End diff --

@pvillard31 : okay got it. Will make this change. 


---

[jira] [Commented] (NIFI-4185) Add XML record reader & writer services

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419507#comment-16419507
 ] 

ASF GitHub Bot commented on NIFI-4185:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2587
  
Thanks for pinging me @JohannesDaniel - I'll also try to have a look over 
the WE / next week. It'd be a great addition!


> Add XML record reader & writer services
> ---
>
> Key: NIFI-4185
> URL: https://issues.apache.org/jira/browse/NIFI-4185
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.3.0
>Reporter: Andy LoPresto
>Assignee: Johannes Peter
>Priority: Major
>  Labels: json, records, xml
>
> With the addition of the {{RecordReader}} and {{RecordSetWriter}} paradigm, 
> XML conversion has not yet been targeted. This will replace the previous 
> ticket for XML to JSON conversion. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2587: NIFI-4185 Add XML Record Reader

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2587
  
Thanks for pinging me @JohannesDaniel - I'll also try to have a look over 
the WE / next week. It'd be a great addition!


---

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419492#comment-16419492
 ] 

Otto Fowler commented on NIFI-3576:
---

[~JPercivall] thoughts?

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5017) ConvertExcelToCSVProcessor should Support Expression Language for Number of Rows to Skip and Columns To Skip

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419493#comment-16419493
 ] 

ASF GitHub Bot commented on NIFI-5017:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178135297
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessorTest.java
 ---
@@ -187,6 +213,34 @@ public void testSkipColumns() throws Exception {
 "9.8765E+08,\n");
 }
 
+@Test
+public void testSkipColumnsWithEL() throws Exception {
+testRunner.enqueue(new 
File("src/test/resources/dataformatting.xlsx").toPath());
--- End diff --

I'm fine with the tests you added. I just suggest that instead of directly 
setting the EL expression in the property value, you do a check against 
attributes that would come with the incoming flow file. Something like:

java
Map attributes = new HashMap();
attributes.put("columnsToSkip", "2");
testRunner.enqueue(new 
File("src/test/resources/dataformatting.xlsx").toPath(), variables);
testRunner.setProperty(ConvertExcelToCSVProcessor.COLUMNS_TO_SKIP, 
"${columnsToSkip}");

This way we check that EL is correctly evaluated against the attributes of 
the incoming flow files.


> ConvertExcelToCSVProcessor  should Support Expression Language for Number of 
> Rows to Skip and Columns To Skip
> -
>
> Key: NIFI-5017
> URL: https://issues.apache.org/jira/browse/NIFI-5017
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Arun A K
>Priority: Minor
>
> The ConvertExcelToCSVProcessor should support expression language for fields 
> "Number of Rows to Skip" and "Columns To Skip". These two could be a 
> precomputed attribute or looked up value and need to be configurable via EL. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2597: NIFI-5017 expression language support for rows and ...

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178135297
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessorTest.java
 ---
@@ -187,6 +213,34 @@ public void testSkipColumns() throws Exception {
 "9.8765E+08,\n");
 }
 
+@Test
+public void testSkipColumnsWithEL() throws Exception {
+testRunner.enqueue(new 
File("src/test/resources/dataformatting.xlsx").toPath());
--- End diff --

I'm fine with the tests you added. I just suggest that instead of directly 
setting the EL expression in the property value, you do a check against 
attributes that would come with the incoming flow file. Something like:

java
Map attributes = new HashMap();
attributes.put("columnsToSkip", "2");
testRunner.enqueue(new 
File("src/test/resources/dataformatting.xlsx").toPath(), variables);
testRunner.setProperty(ConvertExcelToCSVProcessor.COLUMNS_TO_SKIP, 
"${columnsToSkip}");

This way we check that EL is correctly evaluated against the attributes of 
the incoming flow files.


---

[jira] [Commented] (MINIFICPP-440) Build & statically link libressl when not available in system

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419416#comment-16419416
 ] 

ASF GitHub Bot commented on MINIFICPP-440:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/289


> Build & statically link libressl when not available in system
> -
>
> Key: MINIFICPP-440
> URL: https://issues.apache.org/jira/browse/MINIFICPP-440
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> In order to target systems where no SSL library is available, we should 
> support building  & statically linking LibreSSL. This will help make MiNiFi - 
> C++ more portable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp pull request #289: MINIFICPP-440 Added LibreSSL as an extern...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/289


---

[jira] [Commented] (NIFI-5017) ConvertExcelToCSVProcessor should Support Expression Language for Number of Rows to Skip and Columns To Skip

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419337#comment-16419337
 ] 

ASF GitHub Bot commented on NIFI-5017:
--

Github user ak-arun commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178116509
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessorTest.java
 ---
@@ -187,6 +213,34 @@ public void testSkipColumns() throws Exception {
 "9.8765E+08,\n");
 }
 
+@Test
+public void testSkipColumnsWithEL() throws Exception {
+testRunner.enqueue(new 
File("src/test/resources/dataformatting.xlsx").toPath());
--- End diff --

@pvillard31 do you want me to re-write the tests into a single method? 
initially there were separate methods for testing the skipRows and skipColumns, 
I just maintained the same format. 


> ConvertExcelToCSVProcessor  should Support Expression Language for Number of 
> Rows to Skip and Columns To Skip
> -
>
> Key: NIFI-5017
> URL: https://issues.apache.org/jira/browse/NIFI-5017
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Arun A K
>Priority: Minor
>
> The ConvertExcelToCSVProcessor should support expression language for fields 
> "Number of Rows to Skip" and "Columns To Skip". These two could be a 
> precomputed attribute or looked up value and need to be configurable via EL. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2597: NIFI-5017 expression language support for rows and ...

[ak-arun]

Github user ak-arun commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178116509
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessorTest.java
 ---
@@ -187,6 +213,34 @@ public void testSkipColumns() throws Exception {
 "9.8765E+08,\n");
 }
 
+@Test
+public void testSkipColumnsWithEL() throws Exception {
+testRunner.enqueue(new 
File("src/test/resources/dataformatting.xlsx").toPath());
--- End diff --

@pvillard31 do you want me to re-write the tests into a single method? 
initially there were separate methods for testing the skipRows and skipColumns, 
I just maintained the same format. 


---

[GitHub] nifi pull request #2597: NIFI-5017 expression language support for rows and ...

[ak-arun]

Github user ak-arun commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178115417
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java
 ---
@@ -202,9 +204,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final CSVFormat csvFormat = CSVUtils.createCSVFormat(context);
 
 //Switch to 0 based index
-final int firstRow = context.getProperty(ROWS_TO_SKIP).asInteger() 
- 1;
+final int firstRow = 
Integer.parseInt(context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions().getValue())
 - 1;
--- End diff --

I got what you meant, evaluate the expression across the incoming flowfile 
attribute. I will include the change. 


---

[jira] [Commented] (NIFI-5017) ConvertExcelToCSVProcessor should Support Expression Language for Number of Rows to Skip and Columns To Skip

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419328#comment-16419328
 ] 

ASF GitHub Bot commented on NIFI-5017:
--

Github user ak-arun commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178114923
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java
 ---
@@ -202,9 +204,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final CSVFormat csvFormat = CSVUtils.createCSVFormat(context);
 
 //Switch to 0 based index
-final int firstRow = context.getProperty(ROWS_TO_SKIP).asInteger() 
- 1;
+final int firstRow = 
Integer.parseInt(context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions().getValue())
 - 1;
--- End diff --

The DESIRED_SHEETS is already supporting EL


> ConvertExcelToCSVProcessor  should Support Expression Language for Number of 
> Rows to Skip and Columns To Skip
> -
>
> Key: NIFI-5017
> URL: https://issues.apache.org/jira/browse/NIFI-5017
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Arun A K
>Priority: Minor
>
> The ConvertExcelToCSVProcessor should support expression language for fields 
> "Number of Rows to Skip" and "Columns To Skip". These two could be a 
> precomputed attribute or looked up value and need to be configurable via EL. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5017) ConvertExcelToCSVProcessor should Support Expression Language for Number of Rows to Skip and Columns To Skip

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419330#comment-16419330
 ] 

ASF GitHub Bot commented on NIFI-5017:
--

Github user ak-arun commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178115417
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java
 ---
@@ -202,9 +204,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final CSVFormat csvFormat = CSVUtils.createCSVFormat(context);
 
 //Switch to 0 based index
-final int firstRow = context.getProperty(ROWS_TO_SKIP).asInteger() 
- 1;
+final int firstRow = 
Integer.parseInt(context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions().getValue())
 - 1;
--- End diff --

I got what you meant, evaluate the expression across the incoming flowfile 
attribute. I will include the change. 


> ConvertExcelToCSVProcessor  should Support Expression Language for Number of 
> Rows to Skip and Columns To Skip
> -
>
> Key: NIFI-5017
> URL: https://issues.apache.org/jira/browse/NIFI-5017
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Arun A K
>Priority: Minor
>
> The ConvertExcelToCSVProcessor should support expression language for fields 
> "Number of Rows to Skip" and "Columns To Skip". These two could be a 
> precomputed attribute or looked up value and need to be configurable via EL. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2597: NIFI-5017 expression language support for rows and ...

[ak-arun]

Github user ak-arun commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178114923
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java
 ---
@@ -202,9 +204,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final CSVFormat csvFormat = CSVUtils.createCSVFormat(context);
 
 //Switch to 0 based index
-final int firstRow = context.getProperty(ROWS_TO_SKIP).asInteger() 
- 1;
+final int firstRow = 
Integer.parseInt(context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions().getValue())
 - 1;
--- End diff --

The DESIRED_SHEETS is already supporting EL


---

[jira] [Commented] (NIFI-5017) ConvertExcelToCSVProcessor should Support Expression Language for Number of Rows to Skip and Columns To Skip

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419316#comment-16419316
 ] 

ASF GitHub Bot commented on NIFI-5017:
--

Github user ak-arun commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178113911
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java
 ---
@@ -202,9 +204,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final CSVFormat csvFormat = CSVUtils.createCSVFormat(context);
 
 //Switch to 0 based index
-final int firstRow = context.getProperty(ROWS_TO_SKIP).asInteger() 
- 1;
+final int firstRow = 
Integer.parseInt(context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions().getValue())
 - 1;
--- End diff --

Okay, will take care. 


> ConvertExcelToCSVProcessor  should Support Expression Language for Number of 
> Rows to Skip and Columns To Skip
> -
>
> Key: NIFI-5017
> URL: https://issues.apache.org/jira/browse/NIFI-5017
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Arun A K
>Priority: Minor
>
> The ConvertExcelToCSVProcessor should support expression language for fields 
> "Number of Rows to Skip" and "Columns To Skip". These two could be a 
> precomputed attribute or looked up value and need to be configurable via EL. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2597: NIFI-5017 expression language support for rows and ...

[ak-arun]

Github user ak-arun commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178113911
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java
 ---
@@ -202,9 +204,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final CSVFormat csvFormat = CSVUtils.createCSVFormat(context);
 
 //Switch to 0 based index
-final int firstRow = context.getProperty(ROWS_TO_SKIP).asInteger() 
- 1;
+final int firstRow = 
Integer.parseInt(context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions().getValue())
 - 1;
--- End diff --

Okay, will take care. 


---

[GitHub] nifi pull request #2597: NIFI-5017 expression language support for rows and ...

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178112354
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessorTest.java
 ---
@@ -187,6 +213,34 @@ public void testSkipColumns() throws Exception {
 "9.8765E+08,\n");
 }
 
+@Test
+public void testSkipColumnsWithEL() throws Exception {
+testRunner.enqueue(new 
File("src/test/resources/dataformatting.xlsx").toPath());
--- End diff --

To test the EL evaluation againt attributes of the flow file, you can pass 
a Map representing the flow file attributes in the enqueue() 
method.


---

[GitHub] nifi pull request #2597: NIFI-5017 expression language support for rows and ...

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178112075
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java
 ---
@@ -202,9 +204,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final CSVFormat csvFormat = CSVUtils.createCSVFormat(context);
 
 //Switch to 0 based index
-final int firstRow = context.getProperty(ROWS_TO_SKIP).asInteger() 
- 1;
+final int firstRow = 
Integer.parseInt(context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions().getValue())
 - 1;
--- End diff --

Since you're on this... could also be nice to do the EL evaluation against 
the incoming flow file on DESIRED_SHEETS few lines above...


---

[jira] [Commented] (NIFI-5017) ConvertExcelToCSVProcessor should Support Expression Language for Number of Rows to Skip and Columns To Skip

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419309#comment-16419309
 ] 

ASF GitHub Bot commented on NIFI-5017:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178111781
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java
 ---
@@ -202,9 +204,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final CSVFormat csvFormat = CSVUtils.createCSVFormat(context);
 
 //Switch to 0 based index
-final int firstRow = context.getProperty(ROWS_TO_SKIP).asInteger() 
- 1;
+final int firstRow = 
Integer.parseInt(context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions().getValue())
 - 1;
 final String[] sColumnsToSkip = StringUtils
-.split(context.getProperty(COLUMNS_TO_SKIP).getValue(), 
",");
+
.split(context.getProperty(COLUMNS_TO_SKIP).evaluateAttributeExpressions().getValue(),
 ",");
--- End diff --

java

.split(context.getProperty(COLUMNS_TO_SKIP).evaluateAttributeExpressions(flowFile).getValue(),
 ",");



> ConvertExcelToCSVProcessor  should Support Expression Language for Number of 
> Rows to Skip and Columns To Skip
> -
>
> Key: NIFI-5017
> URL: https://issues.apache.org/jira/browse/NIFI-5017
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Arun A K
>Priority: Minor
>
> The ConvertExcelToCSVProcessor should support expression language for fields 
> "Number of Rows to Skip" and "Columns To Skip". These two could be a 
> precomputed attribute or looked up value and need to be configurable via EL. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5017) ConvertExcelToCSVProcessor should Support Expression Language for Number of Rows to Skip and Columns To Skip

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419308#comment-16419308
 ] 

ASF GitHub Bot commented on NIFI-5017:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178111708
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java
 ---
@@ -202,9 +204,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final CSVFormat csvFormat = CSVUtils.createCSVFormat(context);
 
 //Switch to 0 based index
-final int firstRow = context.getProperty(ROWS_TO_SKIP).asInteger() 
- 1;
+final int firstRow = 
Integer.parseInt(context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions().getValue())
 - 1;
--- End diff --

I'd do:
java
final int firstRow = 
context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions(flowFile).asInteger()
 - 1;

Note that I use the EL evaluation against the flow file, otherwise you 
won't look at the incoming FF attributes.


> ConvertExcelToCSVProcessor  should Support Expression Language for Number of 
> Rows to Skip and Columns To Skip
> -
>
> Key: NIFI-5017
> URL: https://issues.apache.org/jira/browse/NIFI-5017
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Arun A K
>Priority: Minor
>
> The ConvertExcelToCSVProcessor should support expression language for fields 
> "Number of Rows to Skip" and "Columns To Skip". These two could be a 
> precomputed attribute or looked up value and need to be configurable via EL. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5017) ConvertExcelToCSVProcessor should Support Expression Language for Number of Rows to Skip and Columns To Skip

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419310#comment-16419310
 ] 

ASF GitHub Bot commented on NIFI-5017:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178112354
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessorTest.java
 ---
@@ -187,6 +213,34 @@ public void testSkipColumns() throws Exception {
 "9.8765E+08,\n");
 }
 
+@Test
+public void testSkipColumnsWithEL() throws Exception {
+testRunner.enqueue(new 
File("src/test/resources/dataformatting.xlsx").toPath());
--- End diff --

To test the EL evaluation againt attributes of the flow file, you can pass 
a Map representing the flow file attributes in the enqueue() 
method.


> ConvertExcelToCSVProcessor  should Support Expression Language for Number of 
> Rows to Skip and Columns To Skip
> -
>
> Key: NIFI-5017
> URL: https://issues.apache.org/jira/browse/NIFI-5017
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Arun A K
>Priority: Minor
>
> The ConvertExcelToCSVProcessor should support expression language for fields 
> "Number of Rows to Skip" and "Columns To Skip". These two could be a 
> precomputed attribute or looked up value and need to be configurable via EL. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5017) ConvertExcelToCSVProcessor should Support Expression Language for Number of Rows to Skip and Columns To Skip

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419311#comment-16419311
 ] 

ASF GitHub Bot commented on NIFI-5017:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178112075
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java
 ---
@@ -202,9 +204,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final CSVFormat csvFormat = CSVUtils.createCSVFormat(context);
 
 //Switch to 0 based index
-final int firstRow = context.getProperty(ROWS_TO_SKIP).asInteger() 
- 1;
+final int firstRow = 
Integer.parseInt(context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions().getValue())
 - 1;
--- End diff --

Since you're on this... could also be nice to do the EL evaluation against 
the incoming flow file on DESIRED_SHEETS few lines above...


> ConvertExcelToCSVProcessor  should Support Expression Language for Number of 
> Rows to Skip and Columns To Skip
> -
>
> Key: NIFI-5017
> URL: https://issues.apache.org/jira/browse/NIFI-5017
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Arun A K
>Priority: Minor
>
> The ConvertExcelToCSVProcessor should support expression language for fields 
> "Number of Rows to Skip" and "Columns To Skip". These two could be a 
> precomputed attribute or looked up value and need to be configurable via EL. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2597: NIFI-5017 expression language support for rows and ...

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178111708
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java
 ---
@@ -202,9 +204,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final CSVFormat csvFormat = CSVUtils.createCSVFormat(context);
 
 //Switch to 0 based index
-final int firstRow = context.getProperty(ROWS_TO_SKIP).asInteger() 
- 1;
+final int firstRow = 
Integer.parseInt(context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions().getValue())
 - 1;
--- End diff --

I'd do:
java
final int firstRow = 
context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions(flowFile).asInteger()
 - 1;

Note that I use the EL evaluation against the flow file, otherwise you 
won't look at the incoming FF attributes.


---

[GitHub] nifi pull request #2597: NIFI-5017 expression language support for rows and ...

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2597#discussion_r178111781
  
--- Diff: 
nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java
 ---
@@ -202,9 +204,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final CSVFormat csvFormat = CSVUtils.createCSVFormat(context);
 
 //Switch to 0 based index
-final int firstRow = context.getProperty(ROWS_TO_SKIP).asInteger() 
- 1;
+final int firstRow = 
Integer.parseInt(context.getProperty(ROWS_TO_SKIP).evaluateAttributeExpressions().getValue())
 - 1;
 final String[] sColumnsToSkip = StringUtils
-.split(context.getProperty(COLUMNS_TO_SKIP).getValue(), 
",");
+
.split(context.getProperty(COLUMNS_TO_SKIP).evaluateAttributeExpressions().getValue(),
 ",");
--- End diff --

java

.split(context.getProperty(COLUMNS_TO_SKIP).evaluateAttributeExpressions(flowFile).getValue(),
 ",");



---

[jira] [Commented] (NIFI-5017) ConvertExcelToCSVProcessor should Support Expression Language for Number of Rows to Skip and Columns To Skip

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419294#comment-16419294
 ] 

ASF GitHub Bot commented on NIFI-5017:
--

Github user ak-arun commented on the issue:

https://github.com/apache/nifi/pull/2597
  
Unit test failures are unrelated


> ConvertExcelToCSVProcessor  should Support Expression Language for Number of 
> Rows to Skip and Columns To Skip
> -
>
> Key: NIFI-5017
> URL: https://issues.apache.org/jira/browse/NIFI-5017
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Arun A K
>Priority: Minor
>
> The ConvertExcelToCSVProcessor should support expression language for fields 
> "Number of Rows to Skip" and "Columns To Skip". These two could be a 
> precomputed attribute or looked up value and need to be configurable via EL. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2597: NIFI-5017 expression language support for rows and columns...

[ak-arun]

Github user ak-arun commented on the issue:

https://github.com/apache/nifi/pull/2597
  
Unit test failures are unrelated


---

[jira] [Commented] (NIFI-5017) ConvertExcelToCSVProcessor should Support Expression Language for Number of Rows to Skip and Columns To Skip

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419290#comment-16419290
 ] 

ASF GitHub Bot commented on NIFI-5017:
--

GitHub user ak-arun opened a pull request:

https://github.com/apache/nifi/pull/2597

NIFI-5017 expression language support for rows and columns to skip

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ak-arun/nifi-1 NIFI-5017

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2597.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2597


commit 7b93fd132988feeb3ce02e5db396775648e69c9c
Author: Arun A K 
Date:   2018-03-29T16:21:05Z

NIFI-5017 expression language support for rows and columns to skip




> ConvertExcelToCSVProcessor  should Support Expression Language for Number of 
> Rows to Skip and Columns To Skip
> -
>
> Key: NIFI-5017
> URL: https://issues.apache.org/jira/browse/NIFI-5017
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Arun A K
>Priority: Minor
>
> The ConvertExcelToCSVProcessor should support expression language for fields 
> "Number of Rows to Skip" and "Columns To Skip". These two could be a 
> precomputed attribute or looked up value and need to be configurable via EL. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (NIFI-5032) Use explicit versions for transitive dependencies

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt resolved NIFI-5032.
---
   Resolution: Fixed
Fix Version/s: 1.7.0

+1 merged to master

> Use explicit versions for transitive dependencies
> -
>
> Key: NIFI-5032
> URL: https://issues.apache.org/jira/browse/NIFI-5032
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
> Fix For: 1.7.0
>
>
> The openid-connect library oauth2-oidc-sdk contains a number of dependencies 
> which have version ranges. These ranges can be problematic to ensuring 
> repeatable builds in different environments. These ranges should be 
> overridden to ensure consistent classpaths. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2597: NIFI-5017 expression language support for rows and ...

[ak-arun]

GitHub user ak-arun opened a pull request:

https://github.com/apache/nifi/pull/2597

NIFI-5017 expression language support for rows and columns to skip

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ak-arun/nifi-1 NIFI-5017

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2597.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2597


commit 7b93fd132988feeb3ce02e5db396775648e69c9c
Author: Arun A K 
Date:   2018-03-29T16:21:05Z

NIFI-5017 expression language support for rows and columns to skip




---

[jira] [Commented] (NIFI-5032) Use explicit versions for transitive dependencies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419284#comment-16419284
 ] 

ASF GitHub Bot commented on NIFI-5032:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2596


> Use explicit versions for transitive dependencies
> -
>
> Key: NIFI-5032
> URL: https://issues.apache.org/jira/browse/NIFI-5032
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>
> The openid-connect library oauth2-oidc-sdk contains a number of dependencies 
> which have version ranges. These ranges can be problematic to ensuring 
> repeatable builds in different environments. These ranges should be 
> overridden to ensure consistent classpaths. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2596: NIFI-5032: Overriding version ranges of transitive ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2596


---

[jira] [Commented] (NIFI-5032) Use explicit versions for transitive dependencies

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419282#comment-16419282
 ] 

ASF subversion and git services commented on NIFI-5032:
---

Commit c4dcde837e10663ced280b99bd64dc1409f951ee in nifi's branch 
refs/heads/master from [~mcgilman]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=c4dcde8 ]

NIFI-5032: This closes #2596.
- Overriding version ranges of transitive dependencies through oauth2-oidc-sdk.

Signed-off-by: joewitt 


> Use explicit versions for transitive dependencies
> -
>
> Key: NIFI-5032
> URL: https://issues.apache.org/jira/browse/NIFI-5032
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>
> The openid-connect library oauth2-oidc-sdk contains a number of dependencies 
> which have version ranges. These ranges can be problematic to ensuring 
> repeatable builds in different environments. These ranges should be 
> overridden to ensure consistent classpaths. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419194#comment-16419194
 ] 

Otto Fowler commented on NIFI-3576:
---

OK, I'll take a stab at it

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler reassigned NIFI-3576:
-

Assignee: Otto Fowler

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Luis Pallares (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419185#comment-16419185
 ] 

Luis Pallares commented on NIFI-3576:
-

true

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419170#comment-16419170
 ] 

Otto Fowler commented on NIFI-3576:
---

So, you want to route the original to a new relationship?  Like invokeHttp does 
with the request relationship?

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Luis Pallares (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419160#comment-16419160
 ] 

Luis Pallares commented on NIFI-3576:
-

>would every flow file have a 'es.hits.total' and maybe a 'es.hit.number'?

yes, 'es.hits.total' to keep the ES convention for this field.

> just empty to a new relationship?
it would be great to have a "not found" relationship when 'es.hits.total' is 0, 
if not, I bet developers will be using an "evaluator" connected to a "router" 
to make it usable, hence my advice to use a new rel to have the (if not results 
then) implicitly done by the processor.

> What would the flow file output be?
> empty to success with the attributes?

flowfile is OK as it is, later we can work on a QueryElasticsearchRecord now 
that everything trends to the record-oriented methodology which is great.

thanks.

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5032) Use explicit versions for transitive dependencies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419055#comment-16419055
 ] 

ASF GitHub Bot commented on NIFI-5032:
--

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/2596

NIFI-5032: Overriding version ranges of transitive dependencies

NIFI-5032:
- Overriding version ranges of transitive dependencies through 
oauth2-oidc-sdk.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-5032

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2596.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2596


commit 52d88ef2633b91f57840002e9651436f7b2168d6
Author: Matt Gilman 
Date:   2018-03-29T14:07:17Z

NIFI-5032:
- Overriding version ranges of transitive dependencies through 
oauth2-oidc-sdk.




> Use explicit versions for transitive dependencies
> -
>
> Key: NIFI-5032
> URL: https://issues.apache.org/jira/browse/NIFI-5032
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>
> The openid-connect library oauth2-oidc-sdk contains a number of dependencies 
> which have version ranges. These ranges can be problematic to ensuring 
> repeatable builds in different environments. These ranges should be 
> overridden to ensure consistent classpaths. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2596: NIFI-5032: Overriding version ranges of transitive ...

[mcgilman]

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/2596

NIFI-5032: Overriding version ranges of transitive dependencies

NIFI-5032:
- Overriding version ranges of transitive dependencies through 
oauth2-oidc-sdk.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-5032

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2596.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2596


commit 52d88ef2633b91f57840002e9651436f7b2168d6
Author: Matt Gilman 
Date:   2018-03-29T14:07:17Z

NIFI-5032:
- Overriding version ranges of transitive dependencies through 
oauth2-oidc-sdk.




---

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418996#comment-16418996
 ] 

Otto Fowler commented on NIFI-3576:
---

So, would every flow file have a 'es.hits.total' and maybe a 'es.hit.number'?

What would the flow file output be? just empty to a new relationship?  or empty 
to success with the attributes?

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Luis Pallares (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418991#comment-16418991
 ] 

Luis Pallares commented on NIFI-3576:
-

I propose, if you are querying something, you might want to know the total hits 
that match with the query search, so no matter if the query matches or not, it 
has to be a success relation response with the "es.hits.total".

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5025) Improve ParseEVTX to maintain state

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418987#comment-16418987
 ] 

Otto Fowler commented on NIFI-5025:
---

What might be better is a more generic component that can do de-duplication 
downstream of this or another processor (most likely by type xml, json)

> Improve ParseEVTX to maintain state
> ---
>
> Key: NIFI-5025
> URL: https://issues.apache.org/jira/browse/NIFI-5025
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Jon Zeolla
>Priority: Major
>
> Currently, ParseEVTX parses Windows EVTX files and sends them without the 
> concept of state/duplication.  In order to use this on our Windows machines 
> we would want such a feature to be more in line with native Windows Event 
> Forwarding capabilities (such as 
> [this|https://docs.microsoft.com/en-us/windows/security/threat-protection/use-windows-event-forwarding-to-assist-in-intrusion-detection#how-is-client-progress-tracked]).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418978#comment-16418978
 ] 

Otto Fowler commented on NIFI-3576:
---

What would the correct behavior be?

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Luis Pallares (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418974#comment-16418974
 ] 

Luis Pallares commented on NIFI-3576:
-

+1

I confirm too, in my use case I can't use a simple InvokeHTTP because I don't 
want to overwrite the original flowfile.

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5032) Use explicit versions for transitive dependencies

[Matt Gilman (JIRA)]

Matt Gilman created NIFI-5032:
-

 Summary: Use explicit versions for transitive dependencies
 Key: NIFI-5032
 URL: https://issues.apache.org/jira/browse/NIFI-5032
 Project: Apache NiFi
  Issue Type: Bug
  Components: Tools and Build
Reporter: Matt Gilman
Assignee: Matt Gilman


The openid-connect library oauth2-oidc-sdk contains a number of dependencies 
which have version ranges. These ranges can be problematic to ensuring 
repeatable builds in different environments. These ranges should be overridden 
to ensure consistent classpaths. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-440) Build & statically link libressl when not available in system

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418966#comment-16418966
 ] 

ASF GitHub Bot commented on MINIFICPP-440:
--

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/289
  
The travis failure is not related. will merge. Thanks


> Build & statically link libressl when not available in system
> -
>
> Key: MINIFICPP-440
> URL: https://issues.apache.org/jira/browse/MINIFICPP-440
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> In order to target systems where no SSL library is available, we should 
> support building  & statically linking LibreSSL. This will help make MiNiFi - 
> C++ more portable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp issue #289: MINIFICPP-440 Added LibreSSL as an external proj...

[phrocker]

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/289
  
The travis failure is not related. will merge. Thanks


---

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418941#comment-16418941
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r178044849
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/AbstractPutHBase.java
 ---
@@ -63,6 +64,17 @@
 .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
 .build();
 
+protected static final PropertyDescriptor DEFAULT_VISIBILITY_STRING = 
new PropertyDescriptor.Builder()
+.name("hbase-default-vis-string")
+.displayName("Default Visibility String")
+.description("When using visibility labels, any value set in 
this field will be applied to all cells that are written unless " +
--- End diff --

It might be worth mentioning that the HBase installation that you're trying 
to write these too must have the VisibilityCoprocessor already loaded. 
Unsuspecting users might just assume that nifi automagically did something to 
HBase to let it handle the labels.


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418940#comment-16418940
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r178043335
  
--- Diff: nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/pom.xml 
---
@@ -79,6 +80,73 @@
 test
 
 
+
+org.apache.hbase
+hbase-server
+1.3.1
+test-jar
+test
+
+
+org.apache.hbase
+hbase-server
+1.3.1
+test
+
+
+
+org.apache.hadoop
+hadoop-hdfs
+2.9.0
+test
+
+
+org.apache.hadoop
+hadoop-hdfs
+test-jar
+2.9.0
+test
+
+
+org.apache.hadoop
+hadoop-common
+2.9.0
+test
+
+
+org.apache.hadoop
+hadoop-common
+2.9.0
+test-jar
+test
+
+
+org.apache.hadoop
+hadoop-auth
+2.9.0
+test
+
+
+org.apache.hbase
+hbase-hadoop-compat
+1.3.1
+test-jar
+test
+
+
+org.apache.hbase
+hbase-hadoop2-compat
+1.3.1
+test-jar
+test
+
+
+org.apache.nifi
+nifi-hbase_1_1_2-client-service
+1.6.0-SNAPSHOT
--- End diff --

Ideally, there's a parent pom with all other nifi modules listed to avoid 
the need to duplicate the version here in the "child" pom.


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418939#comment-16418939
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r178043080
  
--- Diff: nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/pom.xml 
---
@@ -79,6 +80,73 @@
 test
 
 
+
+org.apache.hbase
+hbase-server
+1.3.1
--- End diff --

FYI, 1.3.2 was just released if you want to pick that up :)


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

[joshelser]

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r178044849
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/AbstractPutHBase.java
 ---
@@ -63,6 +64,17 @@
 .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
 .build();
 
+protected static final PropertyDescriptor DEFAULT_VISIBILITY_STRING = 
new PropertyDescriptor.Builder()
+.name("hbase-default-vis-string")
+.displayName("Default Visibility String")
+.description("When using visibility labels, any value set in 
this field will be applied to all cells that are written unless " +
--- End diff --

It might be worth mentioning that the HBase installation that you're trying 
to write these too must have the VisibilityCoprocessor already loaded. 
Unsuspecting users might just assume that nifi automagically did something to 
HBase to let it handle the labels.


---

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

[joshelser]

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r178043080
  
--- Diff: nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/pom.xml 
---
@@ -79,6 +80,73 @@
 test
 
 
+
+org.apache.hbase
+hbase-server
+1.3.1
--- End diff --

FYI, 1.3.2 was just released if you want to pick that up :)


---

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

[joshelser]

Github user joshelser commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r178043335
  
--- Diff: nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/pom.xml 
---
@@ -79,6 +80,73 @@
 test
 
 
+
+org.apache.hbase
+hbase-server
+1.3.1
+test-jar
+test
+
+
+org.apache.hbase
+hbase-server
+1.3.1
+test
+
+
+
+org.apache.hadoop
+hadoop-hdfs
+2.9.0
+test
+
+
+org.apache.hadoop
+hadoop-hdfs
+test-jar
+2.9.0
+test
+
+
+org.apache.hadoop
+hadoop-common
+2.9.0
+test
+
+
+org.apache.hadoop
+hadoop-common
+2.9.0
+test-jar
+test
+
+
+org.apache.hadoop
+hadoop-auth
+2.9.0
+test
+
+
+org.apache.hbase
+hbase-hadoop-compat
+1.3.1
+test-jar
+test
+
+
+org.apache.hbase
+hbase-hadoop2-compat
+1.3.1
+test-jar
+test
+
+
+org.apache.nifi
+nifi-hbase_1_1_2-client-service
+1.6.0-SNAPSHOT
--- End diff --

Ideally, there's a parent pom with all other nifi modules listed to avoid 
the need to duplicate the version here in the "child" pom.


---

[jira] [Commented] (NIFI-5010) putparquet processor in Nifi 1.5 not working

[Bryan Bende (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418842#comment-16418842
 ] 

Bryan Bende commented on NIFI-5010:
---

Have you added anything to NiFi's lib directory, or to the Additional Resources 
property of PutParquet?

> putparquet processor in Nifi 1.5 not working 
> -
>
> Key: NIFI-5010
> URL: https://issues.apache.org/jira/browse/NIFI-5010
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 1.5.0
> Environment: x86_64 GNU/Linux
>Reporter: sujeeth
>Priority: Major
> Attachments: Capture.PNG
>
>
> putpaquet was working in 1.3 version and is not working in 1.5 version after 
> the upgrade.
> I do know that the below mentioned class JniBasedUnixGroupsMapping is 
> available in libarary hadoop-common which i can see under 
> nifi-1.5.0/work/nar/extensions/nifi-hadoop-libraries-nar-1.5.0.nar-unpacked/META-INF/bundled-dependencies/hadoop-common-2.7.3.jar.
>  
> I tried to put this folder in the classpath as well but still no luck
> simple json saving to hdfs using putparquet is failing with below error
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class 
> org.apache.hadoop.security.JniBasedUnixGroupsMapping
>  at java.lang.Class.forName0(Native Method)
>  at java.lang.Class.forName(Class.java:348)
>  at 
> org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$ExtendedConfiguration.getClassByNameOrNull(AbstractHadoopProcessor.java:490)
>  at 
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)
>  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
>  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219)
>  at org.apache.hadoop.security.Groups.(Groups.java:80)
>  at org.apache.hadoop.security.Groups.(Groups.java:74)
>  at 
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
>  at 
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:284)
>  at 
> org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:312)
>  at org.apache.nifi.hadoop.SecurityUtil.loginKerberos(SecurityUtil.java:67)
>  at 
> org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.resetHDFSResources(AbstractHadoopProcessor.java:275)
>  at 
> org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnScheduled(AbstractHadoopProcessor.java:205)
>  at 
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onScheduled(AbstractPutHDFSRecord.java:229)
>  ... 16 common frames omitted



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5031) AvroSchemaRegistry not properly initialized on NiFi restart

[Bryan Bende (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418834#comment-16418834
 ] 

Bryan Bende commented on NIFI-5031:
---

[~junegunn] did this issue happen against a released version or master?

In general, onPropertyModified is called during start up when a component is 
created and given it's properties, so you shouldn't need this fix.

There was an issue on master for a couple of weeks that was reloading a 
component without re-calling onPropertyModified, so  maybe that is what you 
saw, but that is resolved now.

> AvroSchemaRegistry not properly initialized on NiFi restart
> ---
>
> Key: NIFI-5031
> URL: https://issues.apache.org/jira/browse/NIFI-5031
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Junegunn Choi
>Priority: Major
>
> When a NiFi instance is restarted, the internal \{{recordSchemas}} map of 
> AvroSchemaRegistry is not populated. The map is only updated when the content 
> of a dynamic property is updated, so unless we manually update all dynamic 
> properties, schema lookups will fail.
> We should rebuild the map from dynamic properties when the service is 
> enabled. Let me post a patch for this via GitHub pull request.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship

[Wietze B (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418776#comment-16418776
 ] 

Wietze B commented on NIFI-3576:


Are there plans to fix this issue anytime soon? I can confirm the problem is 
still there. I can't think of an easy workaround either, as the flow literally 
stops when Elastic doesn't return results.

> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
> 
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the 
> incoming flowfile and then emits pages of results to the success 
> relationship. If the search returns no results then no pages of results are 
> emitted to the success relationship. 
> The processor should offer other options for handling when there are no 
> results returned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4329) ParseEVTX fails on files containing ForwardedEvents

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418764#comment-16418764
 ] 

Otto Fowler commented on NIFI-4329:
---

Any chance that you can attach a sanitized evtx file that will produce the 
problem?

 

> ParseEVTX fails on files containing ForwardedEvents
> ---
>
> Key: NIFI-4329
> URL: https://issues.apache.org/jira/browse/NIFI-4329
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.1.0, 1.2.0
> Environment: Windows Server 2012, Redhat Linux 6
>Reporter: Colin Dix
>Priority: Major
>  Labels: newbie
>
> When processing EVTX files from the Windows ForwardedEvents channel, the 
> ParseEVTX processor fails with "Bad Chunk". EVTX files from other "normal" 
> channels parse correctly.
> Most recently tested with 1.2.0.3.0.1.0-43.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (NIFI-4407) Non-EL statement processed as expression language

[Otto Fowler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler updated NIFI-4407:
--
Comment: was deleted

(was: I don't think validateExpression or the Range are  documented enough to 
understand what is going on.

in the test:
{code:java}
try {
Query.validateExpression("$${attr}", false);
Assert.fail("invalid query validated");
} catch (final AttributeExpressionLanguageParsingException e) {
}

Query.validateExpression("$${attr}", true);

{code}
the bottom expression doesn't throw or fail, but that is because nothing is 
ever compiled...

 

Can I ask why, when there is a grammar, does nifi manually validate?  It is 
harder when the grammar has one set of rules, then then other rules that might 
be expressed in the grammar are over laid.)

> Non-EL statement processed as expression language
> -
>
> Key: NIFI-4407
> URL: https://issues.apache.org/jira/browse/NIFI-4407
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 1.1.0, 1.2.0, 1.1.1, 1.0.1, 1.3.0
>Reporter: Pierre Villard
>Priority: Critical
>
> If you take a GFF with custom text: {{test$$foo}}
> The generated text will be: {{test$foo}}
> The property supports expression language and one $ is removed during the EL 
> evaluation step. This can be an issue if a user wants to use a value 
> containing to consecutive $$ (such as in password fields).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2592: NIFI-5026: Refactored StandardPreparedQuery so that...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2592


---

[jira] [Commented] (NIFI-5026) Expression Language Query.prepare function can sometimes ignore escape sequence

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418619#comment-16418619
 ] 

ASF subversion and git services commented on NIFI-5026:
---

Commit fd31c161a25fe6009ce35c1e554d4eeb6ed320fd in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=fd31c16 ]

NIFI-5026: Refactored StandardPreparedQuery so that instead of a List of 
Strings that may or may not correspond to compiled expressions and a Map of 
String to Compiled Expression, the StandardPreparedQuery now just takes a List 
of Expression objects, and those Expressions can be evaluated to return the 
proper result

Signed-off-by: Pierre Villard 

This closes #2592.


> Expression Language Query.prepare function can sometimes ignore escape 
> sequence
> ---
>
> Key: NIFI-5026
> URL: https://issues.apache.org/jira/browse/NIFI-5026
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.7.0
>
>
> As reported on the [dev@n.a.o|mailto:dev@n.a.o] mailing list:
>  
> Hi,
> It looks like I have stumbled upon a bug in substitution of evaluated 
> expressions.
> A test case:
> {code:java}
> String result = 
> org.apache.nifi.attribute.expression.language.Query.prepare("${foo}$${foo}").evaluateExpressions(Collections.singletonMap("foo",
>  "bar"), null);{code}
> Expected result:
> {code:java}
> bar${foo}{code}
> Observed result:
> {code:java}
> barbar{code}
> The issue exists in 1.5.0 and, as far as I can tell, in the master branch.The 
> cause is quite simple: Query.prepare(...) splits the input string into pieces 
> to be evaluated and substituted or to be copied literally, but it doesn't 
> keep track of which is which.That couldn't possibly work. If a piece to be 
> copied literally happens to be equal to one of the pieces to be substituted, 
> the things go wrong, as the test case demonstrates.
> Regards,Sergei.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5026) Expression Language Query.prepare function can sometimes ignore escape sequence

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418623#comment-16418623
 ] 

ASF GitHub Bot commented on NIFI-5026:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2592


> Expression Language Query.prepare function can sometimes ignore escape 
> sequence
> ---
>
> Key: NIFI-5026
> URL: https://issues.apache.org/jira/browse/NIFI-5026
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.7.0
>
>
> As reported on the [dev@n.a.o|mailto:dev@n.a.o] mailing list:
>  
> Hi,
> It looks like I have stumbled upon a bug in substitution of evaluated 
> expressions.
> A test case:
> {code:java}
> String result = 
> org.apache.nifi.attribute.expression.language.Query.prepare("${foo}$${foo}").evaluateExpressions(Collections.singletonMap("foo",
>  "bar"), null);{code}
> Expected result:
> {code:java}
> bar${foo}{code}
> Observed result:
> {code:java}
> barbar{code}
> The issue exists in 1.5.0 and, as far as I can tell, in the master branch.The 
> cause is quite simple: Query.prepare(...) splits the input string into pieces 
> to be evaluated and substituted or to be copied literally, but it doesn't 
> keep track of which is which.That couldn't possibly work. If a piece to be 
> copied literally happens to be equal to one of the pieces to be substituted, 
> the things go wrong, as the test case demonstrates.
> Regards,Sergei.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5026) Expression Language Query.prepare function can sometimes ignore escape sequence

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-5026:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Expression Language Query.prepare function can sometimes ignore escape 
> sequence
> ---
>
> Key: NIFI-5026
> URL: https://issues.apache.org/jira/browse/NIFI-5026
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.7.0
>
>
> As reported on the [dev@n.a.o|mailto:dev@n.a.o] mailing list:
>  
> Hi,
> It looks like I have stumbled upon a bug in substitution of evaluated 
> expressions.
> A test case:
> {code:java}
> String result = 
> org.apache.nifi.attribute.expression.language.Query.prepare("${foo}$${foo}").evaluateExpressions(Collections.singletonMap("foo",
>  "bar"), null);{code}
> Expected result:
> {code:java}
> bar${foo}{code}
> Observed result:
> {code:java}
> barbar{code}
> The issue exists in 1.5.0 and, as far as I can tell, in the master branch.The 
> cause is quite simple: Query.prepare(...) splits the input string into pieces 
> to be evaluated and substituted or to be copied literally, but it doesn't 
> keep track of which is which.That couldn't possibly work. If a piece to be 
> copied literally happens to be equal to one of the pieces to be substituted, 
> the things go wrong, as the test case demonstrates.
> Regards,Sergei.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5030) ControlRate stops allowing FlowFiles through from same connection once limit for any 'group' is reached

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418604#comment-16418604
 ] 

ASF GitHub Bot commented on NIFI-5030:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2593


> ControlRate stops allowing FlowFiles through from same connection once limit 
> for any 'group' is reached
> ---
>
> Key: NIFI-5030
> URL: https://issues.apache.org/jira/browse/NIFI-5030
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.7.0
>
>
> If we have FlowFiles in a single connection that feed into ControlRate, and 
> we use the "Grouping Attribute" property, then whenever the processor 
> encounters one FlowFile that should not continue on, subsequent FlowFiles in 
> the queue will also not be transferred on either, even if they are of a 
> different group.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5030) ControlRate stops allowing FlowFiles through from same connection once limit for any 'group' is reached

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-5030:
-
   Resolution: Fixed
Fix Version/s: 1.7.0
   Status: Resolved  (was: Patch Available)

> ControlRate stops allowing FlowFiles through from same connection once limit 
> for any 'group' is reached
> ---
>
> Key: NIFI-5030
> URL: https://issues.apache.org/jira/browse/NIFI-5030
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.7.0
>
>
> If we have FlowFiles in a single connection that feed into ControlRate, and 
> we use the "Grouping Attribute" property, then whenever the processor 
> encounters one FlowFile that should not continue on, subsequent FlowFiles in 
> the queue will also not be transferred on either, even if they are of a 
> different group.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)