davsclaus opened a new pull request, #3226:
URL: https://github.com/apache/camel-kamelets/pull/3226
Part of #3170: `elasticsearch-index-sink` and `opensearch-index-sink` take a
connection and a socket timeout, and their `certificate` now also takes a DER
certificate. `solr-sink` and `solr-source` take a request handler, a connection
timeout and a request timeout.
### The options
| Option | Kamelets | Template mapping | What it does |
|---|---|---|---|
| `connectionTimeout` | `elasticsearch-index-sink`, `opensearch-index-sink`
| `connectionTimeout={{?connectionTimeout}}` | How long to wait for the
connection, in ms. Unset: 30000. |
| `socketTimeout` | `elasticsearch-index-sink`, `opensearch-index-sink` |
`socketTimeout={{?socketTimeout}}` | How long to wait for data from the
cluster, in ms. Unset: 30000. |
| `requestHandler` | `solr-sink`, `solr-source` |
`requestHandler={{?requestHandler}}` | The path of another handler of the
collection, such as `/update/catalog` or `/query`. Unset: `/update` for the
sink, `/select` for the source. It has a `pattern:` (`^/[A-Za-z0-9_./-]+$`). |
| `connectionTimeout` | `solr-sink`, `solr-source` |
`connectionTimeout={{?connectionTimeout}}` | How long to wait for the
connection, in ms. Unset: 60000. |
| `requestTimeout` | `solr-sink`, `solr-source` |
`requestTimeout={{?requestTimeout}}` | How long to wait for the answer of Solr,
in ms. Unset: 600000, which is 10 minutes. |
- **Defaults:** all the new options are optional and have no `default`, as
in #3174. When they are unset, the optional placeholders drop them, and the
endpoint URIs are the same as on `main`.
- **`certificate` on Elasticsearch and OpenSearch:** this option already was
the trust store for a private CA, and it works: the Elasticsearch `http_ca.crt`
and the OpenSearch demo `root-ca.pem` make HTTPS work, and without them the
message fails with `PKIX path building failed`. But it took only PEM text
encoded in base64. A DER certificate encoded in base64 contains `+`, which
became a space, and the integration failed to start with `Illegal base64
character 20`. The mapping is now
`certificatePath=RAW(base64:{{?certificate}})`, the same as #3173 did for Solr,
so both PEM and DER work. The option descriptions now say so, and give the
`base64` command.
- **`size` on `solr-source`:** the number of rows was already added by #3173
as `size`. It is tested again below, and the partial page has an example with
`size: 500`.
- **Descriptions:** the "Good to know" paragraphs say how long a call waits
when it is not answered, and which option changes that. The Elasticsearch and
OpenSearch paragraphs say what to do for a private CA. The Solr paragraphs name
the default handlers. The partial pages have new sections and an example each.
### Left out
- **`maxRetryTimeout`:** in camel-elasticsearch and camel-opensearch 4.24 it
is deprecated and has no effect, as the current REST client ignores it.
- **A trust store file, or no hostname check:** without a bean, the
components take one CA certificate (`certificatePath`), which `certificate`
maps. A JKS or PKCS12 trust store needs an `SSLContextParameters` bean:
`opensearch-index-sink` already takes one in `sslContextParameters`,
`elasticsearch-index-sink` has no such option yet. The hostname verifier of
camel-opensearch is a bean too. With the demo certificates, the host name check
passes, as `esnode.pem` names `localhost` and `127.0.0.1`.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT against Elasticsearch 9.1.0
(security on, HTTPS with the CA it generates), OpenSearch 3.9.0 (HTTPS with the
demo certificates) and Solr 9.10.1, all in Docker. A slow server was made with
toxiproxy (3 s of latency on every packet from the server), a server that never
answers with a local TCP socket that accepts and stays silent, and a connection
that never opens with the address `10.255.255.1`.
| Case | Result |
|---|---|
| Options unset, all four Kamelets, `main` and this branch | `camel get
endpoint --json` shows identical URIs |
| Elasticsearch / OpenSearch, `certificate` = the CA as base64 PEM |
Document `order-42` stored |
| The same, `certificate` = the CA as base64 DER (has `+`), `main` | The
integration does not start: `Illegal base64 character 20` |
| The same, this branch | Document stored, on both |
| Elasticsearch / OpenSearch, no `certificate` | Message fails: `PKIX path
building failed`, on both |
| Elasticsearch with the OpenSearch CA in `certificate` | Message fails:
`PKIX path building failed` (only the given CA is trusted) |
| Elasticsearch, slow server, timeouts unset | Stored after 6.2 s |
| Elasticsearch, slow server, `socketTimeout: 1000` | Fails after 1.1 s:
`SocketTimeoutException: 1.000 milliseconds timeout` |
| Elasticsearch, slow server, `socketTimeout: 5000` | Stored after 6.2 s |
| OpenSearch, slow server, timeouts unset | Stored after 9.2 s |
| OpenSearch, slow server, `socketTimeout: 1000` | Fails after 4.2 s:
`Connection is closed` (the HTTP/2 client closes the connection on the timeout)
|
| OpenSearch, slow server, `socketTimeout: 5000` | Stored after 9.2 s |
| Elasticsearch / OpenSearch, server that never answers, timeouts unset |
Fails after 30 s with `SocketTimeoutException`, on both |
| Elasticsearch / OpenSearch, `10.255.255.1`, `connectionTimeout: 1000` |
Fails after 1.1 s with a connect timeout, on both |
| Elasticsearch / OpenSearch, `10.255.255.1`, timeouts unset | Fails after
30 s with a connect timeout, on both |
| Elasticsearch / OpenSearch, `connectionTimeout: 5000`, `socketTimeout:
7000` | Both appear in the endpoint URI |
| `solr-source`, 25 documents, `size` unset / 20 / 50 | 10 / 20 / 25
documents |
| `solr-source`, `requestHandler: /newOrders` (a search handler with the
invariant `fq=status_s:new`), `size: 50` | 13 documents, only the new ones |
| `solr-source`, `requestHandler: /nothere` | The query fails with
`RemoteSolrException` from `/solr/products/nothere`, and is logged |
| `solr-sink`, `requestHandler` unset | 25 documents stored, none with
`via_s` |
| `solr-sink`, `requestHandler: /update/tagged` (an update handler that adds
`via_s:tagged` with the template processor) | 25 documents stored, all with
`via_s: tagged` |
| `solr-sink`, `requestHandler: /update/nothere` | Message fails with
`RemoteSolrException` |
| `solr-sink`, slow server, timeouts unset / `requestTimeout: 10000` |
Stored after 6.1 s |
| `solr-sink`, slow server, `requestTimeout: 1000` | Fails after 2.1 s:
`HttpTimeoutException: request timed out` |
| `solr-source`, slow server, `requestTimeout: 1000` | Each query fails with
`HttpTimeoutException`, is logged, and the next one runs 15 s later |
| `solr-sink`, server that never answers, timeouts unset | Fails after 20
min: `HttpTimeoutException: request timed out` (a `HEAD` request, then the
request, each waited 10 min) |
| `solr-sink`, server that never answers, `requestTimeout: 3000`, two
messages | The first fails after 6 s, the second after 3 s |
| `solr-source`, server that never answers, `requestTimeout: 3000` | The
query fails after 3 s |
| `solr-sink`, `10.255.255.1`, `connectionTimeout: 1000` | Fails after 2.1
s: `HttpConnectTimeoutException` |
| `solr-sink`, `10.255.255.1`, timeouts unset | Fails after 120 s:
`HttpConnectTimeoutException` |
| `solr-source`, all three options set | `connectionTimeout=5000`,
`requestHandler=/newOrders`, `requestTimeout=7000` and `size=50` in the
endpoint URI |
| The examples of the four partial pages, run against the containers | All
work; the OpenSearch one with a DER CA |
Over plain HTTP, the Solr client checks the server with a `HEAD` request
before its first `POST`, once per client. Each request waits for its own
timeouts, so the first message of `solr-sink` waits up to twice as long, for
example 20 minutes when Solr never answers, or two minutes when it cannot be
reached. The descriptions of `solr-sink` say so. The queries of `solr-source`
are `GET` requests, which wait only once.
`./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from
the root. Before building, the library copies of the four Kamelets were
refreshed with `./mvnw -pl library/camel-kamelets
resources:copy-resources@copy-resource-one`. The build regenerated
`kamelets-index.json`.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]