CrownChu opened a new pull request, #10777:
URL: https://github.com/apache/paimon/pull/10777
### Purpose
Add `paimon_es_search` over Paimon's existing Java Elasticsearch global
indexes, with a shared request and result contract for Spark, Flink and the
Java SDK. Today callers can use the vector/full-text entry points individually;
this change also exposes a bounded Elasticsearch-style DSL, including text
queries, Boolean filters and filtered KNN, with snapshot-consistent source
fetching.
The branch is based directly on `apache/paimon:master`. It contains two
commits:
1. Rename the `paimon-eslib` directory, Maven artifact and opt-in profile to
`paimon-elasticsearch`. Keep the Java packages, `es-index` identifier, table
options and persisted Java archive format compatible.
2. Add the shared request parser/planner/results, the Java Lucene query
adapter, Spark table-valued function, distributed Flink procedure and Java SDK
entry point. Extend the existing Java 11 CI lane to exercise all three entry
points.
Usage:
- Spark: `SELECT * FROM paimon_es_search('catalog', '<request_json>')`.
- Flink: `CALL catalog.sys.paimon_es_search('<request_json>')`.
- Java: `EsIndexSearch.search(catalog, requestJson)`.
The Java integration reuses the existing published ESLib 1.0.7 and Lucene
9.12.0 dependencies. Index construction and persisted metadata are unchanged.
The query adapter owns its archive readers independently and preserves stream
cleanup when loading fails and is retried.
The initial contract supports one append-only data-evolution table, one
ordered `es-index` field group, latest or explicit snapshots, text/scalar/bool
queries and root KNN with an optional prefilter. Queries reject
incomplete/stale index coverage, incompatible field mappings, deletion-vector
snapshots and query authorization policies. Pagination uses a bounded global
score/row-ID ordering; text scores use shard-local statistics. Aggregations,
custom sort, multi-target search and tags are outside this initial contract.
See [the search
guide](https://github.com/CrownChu/paimon/blob/codex/paimon-java-es-search-community-20261023/paimon-elasticsearch/docs/paimon-es-search.html)
for request examples, result decoding, prerequisites and limits.
### Tests
Local validation passed **101 unique functional tests** on JDK 11 with Maven
`verify`, including Checkstyle, Spotless, Enforcer and RAT. The CI
routing/command suite also passed **30 tests**. These are local results;
upstream CI is reported separately by GitHub.
```sh
mvn -B -ntp -Pflink1,spark3,paimon-elasticsearch \
-pl paimon-spark/paimon-spark-ut,paimon-flink/paimon-flink-common -am \
-Dpaimon.es.java.tests -Dmaven.jar.forceCreation=true \
'-Dtest=EsSearch*Test,ESIndex*Test,PaimonEsSearchProcedureITCase' \
-DfailIfNoTests=false \
-DwildcardSuites=org.apache.paimon.spark.catalyst.plans.logical.PaimonEsSearchQueryTest,org.apache.paimon.spark.query.PaimonEsSearchJavaTest
\
verify
```
Coverage includes Java archive compatibility, failed-load cleanup/retry,
HNSW / INT8 HNSW / DiskBBQ, text/bool/filter-KNN, pagination, source reads,
snapshot pinning, stale/incomplete index rejection, field renames, and
Spark/Flink result parity with the Java SDK. Flink tests execute shard work
through a local Flink runtime; Spark tests use `local[2]`.
CI routing/command tests:
```sh
python3 -m unittest discover -s tools/ci -p 'test_*.py'
CI_DRY_RUN=true bash tools/ci/run-java-tests.sh eslib 2.12 11
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
