petercheon commented on issue #67213:
URL: https://github.com/apache/airflow/issues/67213#issuecomment-4493279908
Verified **#66065 — fix(elasticsearch): pin compatible-with at the transport
layer to keep ES 8 servers working** in
`apache-airflow-providers-elasticsearch==6.5.4rc1`. ✅
### Environment
- Airflow `3.1.3` (clean venv, Python 3.11)
- `apache-airflow-providers-elasticsearch==6.5.4rc1` from PyPI
- `elasticsearch-py==9.4.0` (the default >=9 resolution that triggers the
regression)
- Server: live Elasticsearch **8.17.3** cluster
### Steps
1. **Reproduced the regression first** (no fix applied): a raw
`Elasticsearch(...)` client against the ES 8 server fails with the documented
error —
```
BadRequestError(400, 'media_type_header_exception',
Accept version must be either version 8 or 7, but found 9.
Accept=application/vnd.elasticsearch+json; compatible-with=9)
```
2. **Applied the fix** by setting `AIRFLOW__ELASTICSEARCH__ES_COMPAT_WITH=8`
and calling
`airflow.providers.elasticsearch._compat.apply_compat_with(client)`. The
following all succeed against the same ES 8 server:
- `client.info()` → `version.number = 8.17.3`
- `client.cluster.health()` → `status=green`
- `client.search(index="missing")` → clean 404 (no media-type error)
- `client.bulk(operations=...)` → `errors=False`, both docs indexed,
`count=2`
3. **Wire-level confirmation** — swapped
`elastic_transport.Transport.perform_request` for a spy and captured the
headers actually sent:
```
info() Accept : application/vnd.elasticsearch+json;
compatible-with=8
bulk() Accept : application/vnd.elasticsearch+json;
compatible-with=8
bulk() Content-Type : application/vnd.elasticsearch+x-ndjson;
compatible-with=8
```
The `+x-ndjson` Content-Type rewrite (the originally-broken bulk path) is
correctly applied.
4. **End-to-end through the hook**: `ElasticsearchPythonHook` returns a
client whose transport is wrapped when `es_compat_with` is set, and unwrapped
when it isn't — same `hook.search(...)` call succeeds with the option set and
fails with the 400 without it.
Looks good to ship from my side. 🎉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
