David Vassallo created NIFI-6404:
------------------------------------
Summary: PutElasticsearchHttp: Remove _type as being compulsory
Key: NIFI-6404
URL: https://issues.apache.org/jira/browse/NIFI-6404
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Affects Versions: 1.9.2
Environment: Elasticsearch 7.x
Reporter: David Vassallo
In ES 7.x and above, document "type" is no longer compulsory and in fact is
deprecated. When using the 1.9.2 version of PutElasticsearchHttp with ES v7.2,
it still works however you'll see the following HTTP in the response:
{{HTTP/1.1 200 OK}}
{{Warning: 299 Elasticsearch-7.2.0-508c38a "[types removal] Specifying types in
bulk requests is deprecated."}}
{{content-type: application/json; charset=UTF-8}}
The fix is relatively straightforward:
* In *PutElasticserachHttp.java*, remove the requirement of a compulsory
"Type" property:
{{public static final PropertyDescriptor TYPE = new
PropertyDescriptor.Builder()}}
{{ .name("put-es-type")}}
{{ .displayName("Type")}}
{{ .description(*"The type of this document (used by Elasticsearch < 7.0 for
indexing and searching). Leave empty for ES >= 7.0"*)}}
{{ *.required(false)*}}
{{ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)}}
{{ .addValidator(StandardValidators.NON_EMPTY_EL_VALIDATOR)}}
{{ .build();}}
* In *AbstractElasticsearchHttpProcessor.java*, check for the presence of
"docType". If not present, assume elasticsearch 7.x or above and omit from bulk
API URL:
{{protected void buildBulkCommand(StringBuilder sb, String index, String
docType, String indexOp, String id, String jsonString) {}}
{{ if (indexOp.equalsIgnoreCase("index")) {}}
{{ sb.append("{\"index\": { \"_index\": \"");}}
{{ sb.append(StringEscapeUtils.escapeJson(index));}}
{{ *if (!(StringUtils.isEmpty(docType) | docType == null)){*}}
{{ *sb.append("\", \"_type\": \"");*}}
{{ *sb.append(StringEscapeUtils.escapeJson(docType));*}}
{{ *sb.append("\"");*}}
{{ *}*}}
{{ if (!StringUtils.isEmpty(id)) {}}
{{ sb.append(", \"_id\": \"");}}
{{ sb.append(StringEscapeUtils.escapeJson(id));}}
{{ sb.append("\"");}}
{{ }}}
{{ sb.append("}}\n");}}
{{ sb.append(jsonString);}}
{{ sb.append("\n");}}
{{ } else if (indexOp.equalsIgnoreCase("upsert") ||
indexOp.equalsIgnoreCase("update")) {}}
{{ sb.append("{\"update\": { \"_index\": \"");}}
{{ sb.append(StringEscapeUtils.escapeJson(index));}}
{{ sb.append("\", \"_type\": \"");}}
{{ sb.append(StringEscapeUtils.escapeJson(docType));}}
{{ sb.append("\", \"_id\": \"");}}
{{ sb.append(StringEscapeUtils.escapeJson(id));}}
{{ sb.append("\" }\n");}}
{{ sb.append("{\"doc\": ");}}
{{ sb.append(jsonString);}}
{{ sb.append(", \"doc_as_upsert\": ");}}
{{ sb.append(indexOp.equalsIgnoreCase("upsert"));}}
{{ sb.append(" }\n");}}
{{ } else if (indexOp.equalsIgnoreCase("delete")) {}}
{{ sb.append("{\"delete\": { \"_index\": \"");}}
{{ sb.append(StringEscapeUtils.escapeJson(index));}}
{{ sb.append("\", \"_type\": \"");}}
{{ sb.append(StringEscapeUtils.escapeJson(docType));}}
{{ sb.append("\", \"_id\": \"");}}
{{ sb.append(StringEscapeUtils.escapeJson(id));}}
{{ sb.append("\" }\n");}}
{{ }}}
{{}}}
* The *TestPutElasticsearchHttp.java* test file needs to be updated to reflect
that now a requests without type is valid (it's currently marked as invalid)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
