MikeThomsen commented on code in PR #6544:
URL: https://github.com/apache/nifi/pull/6544#discussion_r1012410138
##########
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java:
##########
@@ -416,6 +495,26 @@ public void refresh(final String index, final Map<String,
String> requestParamet
}
}
+ @Override
+ public boolean exists(final String index, final Map<String, String>
requestParameters) {
+ try {
+ final Response response = performRequest("HEAD", "/" + index,
requestParameters, null);
Review Comment:
Should probably check to make sure that `index` has no leading or trailing
`/`
##########
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-test-utils/pom.xml:
##########
@@ -0,0 +1,70 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Licensed to the Apache Software Foundation (ASF) under one or more
contributor
+license agreements. See the NOTICE file distributed with this work for
additional
+information regarding copyright ownership. The ASF licenses this file to
+You under the Apache License, Version 2.0 (the "License"); you may not use
+this file except in compliance with the License. You may obtain a copy of
+the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
+by applicable law or agreed to in writing, software distributed under the
+License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
+OF ANY KIND, either express or implied. See the License for the specific
+language governing permissions and limitations under the License. -->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
https://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <artifactId>nifi-elasticsearch-bundle</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-elasticsearch-test-utils</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.elasticsearch.client</groupId>
+ <artifactId>elasticsearch-rest-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>testcontainers</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>elasticsearch</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
Review Comment:
Should be made into a compile time dependency w/ a version number.
##########
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ElasticsearchRestProcessor.java:
##########
@@ -117,13 +123,66 @@ default String getQuery(final FlowFile input, final
ProcessContext context, fina
}
default Map<String, String> getUrlQueryParameters(final ProcessContext
context, final FlowFile flowFile) {
+ return getUrlQueryParameters(context, flowFile != null ?
flowFile.getAttributes() : null);
+ }
+
+ default Map<String, String> getUrlQueryParameters(final ProcessContext
context, final Map<String, String> attributes) {
return context.getProperties().entrySet().stream()
// filter non-null dynamic properties
.filter(e -> e.getKey().isDynamic() && e.getValue() != null)
// convert to Map of URL parameter keys and values
.collect(Collectors.toMap(
- e -> e.getKey().getName(),
- e ->
context.getProperty(e.getKey()).evaluateAttributeExpressions(flowFile).getValue()
+ e -> e.getKey().getName(),
+ e ->
context.getProperty(e.getKey()).evaluateAttributeExpressions(attributes).getValue()
));
}
+
+ String VERIFICATION_STEP_INDEX_EXISTS = "Elasticsearch Index Exists";
+
+ @Override
+ default List<ConfigVerificationResult> verify(final ProcessContext
context, final ComponentLog verificationLogger, final Map<String, String>
attributes) {
Review Comment:
I have reservations about this approach for verification because it's very
possible users could end up using EL to specify a dynamic index name based on
an Elastic template. How do you see that scenario playing out with verification
since the index may not exist yet for a good reason? Also, this could
potentially block users who are iterating quickly and relying on Elastic to
autogenerate the schema during their initial setup before formalizing one.
##########
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/GetElasticsearch.java:
##########
@@ -213,7 +273,7 @@ private void handleElasticsearchException(final
ElasticsearchException ese, Flow
getLogger().error(msg, ese);
if (input != null) {
session.penalize(input);
- input = session.putAttribute(input, "elasticsearch.get.error",
ese.getMessage());
+ session.putAttribute(input, "elasticsearch.get.error",
ese.getMessage());
Review Comment:
Why did you drop the assignment?
##########
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ElasticsearchRestProcessor.java:
##########
@@ -117,13 +123,66 @@ default String getQuery(final FlowFile input, final
ProcessContext context, fina
}
default Map<String, String> getUrlQueryParameters(final ProcessContext
context, final FlowFile flowFile) {
+ return getUrlQueryParameters(context, flowFile != null ?
flowFile.getAttributes() : null);
+ }
+
+ default Map<String, String> getUrlQueryParameters(final ProcessContext
context, final Map<String, String> attributes) {
return context.getProperties().entrySet().stream()
// filter non-null dynamic properties
.filter(e -> e.getKey().isDynamic() && e.getValue() != null)
// convert to Map of URL parameter keys and values
.collect(Collectors.toMap(
- e -> e.getKey().getName(),
- e ->
context.getProperty(e.getKey()).evaluateAttributeExpressions(flowFile).getValue()
+ e -> e.getKey().getName(),
+ e ->
context.getProperty(e.getKey()).evaluateAttributeExpressions(attributes).getValue()
));
}
+
+ String VERIFICATION_STEP_INDEX_EXISTS = "Elasticsearch Index Exists";
+
+ @Override
+ default List<ConfigVerificationResult> verify(final ProcessContext
context, final ComponentLog verificationLogger, final Map<String, String>
attributes) {
+ final List<ConfigVerificationResult> results = new ArrayList<>();
+ final ConfigVerificationResult.Builder indexExistsResult = new
ConfigVerificationResult.Builder()
+ .verificationStepName(VERIFICATION_STEP_INDEX_EXISTS);
+
+ ElasticSearchClientService verifyClientService = null;
+ String index = null;
+ boolean indexExists = false;
+ if (context.getProperty(CLIENT_SERVICE).isSet()) {
+ verifyClientService =
context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ if (context.getProperty(INDEX).isSet()) {
+ index =
context.getProperty(INDEX).evaluateAttributeExpressions(attributes).getValue();
+ try {
+ if (verifyClientService.exists(index,
getUrlQueryParameters(context, attributes))) {
+
indexExistsResult.outcome(ConfigVerificationResult.Outcome.SUCCESSFUL)
+ .explanation(String.format("Index [%s]
exists", index));
+ indexExists = true;
+ } else {
+
indexExistsResult.outcome(ConfigVerificationResult.Outcome.SUCCESSFUL)
Review Comment:
Related to the point above, I guess the goal here is to just flex the REST
api and make sure you can get an expected response (even if it's an index
doesn't exist exception)?
##########
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-test-utils/pom.xml:
##########
@@ -0,0 +1,70 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Licensed to the Apache Software Foundation (ASF) under one or more
contributor
+license agreements. See the NOTICE file distributed with this work for
additional
+information regarding copyright ownership. The ASF licenses this file to
+You under the Apache License, Version 2.0 (the "License"); you may not use
+this file except in compliance with the License. You may obtain a copy of
+the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
+by applicable law or agreed to in writing, software distributed under the
+License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
+OF ANY KIND, either express or implied. See the License for the specific
+language governing permissions and limitations under the License. -->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
https://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <artifactId>nifi-elasticsearch-bundle</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-elasticsearch-test-utils</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.elasticsearch.client</groupId>
+ <artifactId>elasticsearch-rest-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>testcontainers</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>elasticsearch</artifactId>
Review Comment:
Should be made into a compile time dependency w/ a version number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
