Akash3121 commented on code in PR #10777: URL: https://github.com/apache/paimon/pull/10777#discussion_r4369566359
########## paimon-elasticsearch/docs/paimon-es-search.html: ########## @@ -0,0 +1,178 @@ +<!doctype html> +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +<html lang="en"> +<head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>paimon_es_search: Java search for Spark, Flink and SDK</title> + <style> + body { max-width: 960px; margin: 40px auto; padding: 0 24px; color: #202a35; + font: 16px/1.65 system-ui, sans-serif; } + h1, h2 { line-height: 1.25; } + h2 { margin-top: 36px; } + code, pre { font-family: ui-monospace, monospace; font-size: .9em; } + pre { overflow-x: auto; padding: 20px; background: #f1f5f8; border-radius: 6px; } + table { border-collapse: collapse; width: 100%; } + td, th { padding: 10px; border: 1px solid #d6dfe7; text-align: left; vertical-align: top; } + a { color: #1459a6; } + </style> +</head> +<body> +<h1>Query ESM1 indexes with <code>paimon_es_search</code></h1> +<p>Spark, Flink and the Java SDK share the same version-1 request, snapshot planning, +field-ID binding, global reduction and result records. The Java module builds and queries Lucene archives using the published ESLib dependency.</p> +<table><tr><th>Entry</th><th>Execution</th></tr> +<tr><td>Spark <code>paimon_es_search(catalog, request_json)</code></td><td>Shard queries on executors; bounded global top-K and source fetch on driver.</td></tr> +<tr><td>Flink <code>CALL catalog.sys.paimon_es_search(request_json)</code></td><td>Shard groups on TaskManagers; bounded worker top-K followed by client merge/source fetch. Returns one JSON string per HIT/PAGE_END row.</td></tr> +<tr><td>Java <code>EsIndexSearch.search(catalog, requestJson)</code></td><td>Local SDK execution with the same planning, readers and bounded result.</td></tr></table> +<h2>Runtime and index prerequisites</h2> +<p>Use JDK 11+, the matching Paimon connector and <code>org.apache.paimon:paimon-elasticsearch</code> +with its POM runtime dependencies (published ESLib 1.0.7 and Lucene 9.12.0). The former module, +artifact and profile were named <code>paimon-eslib</code>; Java package names and <code>es-index</code> +remain unchanged. Keep the connector, index module and dependencies together on every worker. +IK requires an additional compatible analyzer JAR.</p> +<p>The existing Java archive format and persisted field mappings remain compatible. +The module rename does not require an index rebuild.</p> +<p>Use an append-only data-evolution table with tracked row IDs and construction-snapshot metadata +emitted by <code>create_global_index</code>. Spark additionally needs +<code>org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions</code>. No Elasticsearch server is required for this search entry.</p> +<pre><code>CREATE DATABASE IF NOT EXISTS paimon.demo; +CREATE TABLE paimon.demo.docs ( + id INT, embedding ARRAY<FLOAT>, title STRING, tag STRING +) USING paimon TBLPROPERTIES ( + 'bucket'='-1', + 'row-tracking.enabled'='true', + 'data-evolution.enabled'='true', + 'global-index.es-index.fields.embedding.dimension'='3', + 'global-index.es-index.fields.embedding.algorithm'='hnsw', + 'global-index.es-index.fields.embedding.metric'='l2', + 'global-index.es-index.fields.title.type'='fulltext', + 'global-index.es-index.fields.tag.type'='keyword' +); +INSERT INTO paimon.demo.docs VALUES + (1, array(1.0f, 0.0f, 0.0f), 'paimon lake', 'active'), + (2, array(0.0f, 1.0f, 0.0f), 'river meadow', 'inactive'); +CALL paimon.sys.create_global_index( + table => 'demo.docs', index_column => 'embedding,title,tag,id', index_type => 'es-index' +);</code></pre> +<p>The new schema above has field IDs <code>id=0, embedding=1, title=2, tag=3</code>. +For an existing table, read its schema IDs; do not infer them from current column positions. +The index field group must match the persisted order. Query aliases bind to these stable IDs.</p> + +<h2>Text and filter-KNN examples</h2> +<pre><code>SELECT row_id, score, rank, source_schema_json, source_binary +FROM paimon_es_search('paimon', '{ + "version": 1, "operation": "SEARCH", + "targets": [{ + "name": "docs", "database": "demo", "table": "docs", + "indexes": [[1, 2, 3, 0]], + "bindings": {"text": 2, "label": 3, "v": 1, "number": 0} + }], + "request": { + "query": {"match": {"text": "paimon"}}, + "size": 10, "fetch": "SOURCE" + } +}') WHERE record_kind = 'HIT' ORDER BY rank;</code></pre> +<p>To run filter-KNN, replace the <code>request</code> object with:</p> +<pre><code>{ + "query": {"knn": { + "field": "v", "query_vector": [1.0, 0.0, 0.0], + "k": 3, "num_candidates": 100, + "filter": {"term": {"label": "active"}} + }}, + "size": 3, "fetch": "SOURCE" +}</code></pre> +<p>Put vector prefilters inside <code>knn.filter</code>. An outer SQL <code>WHERE</code> +filters the returned result rows and does not change the KNN candidate set.</p> + +<h2>Supported contract</h2> +<table> + <tr><th>Area</th><th>Behavior</th></tr> + <tr><td>Arguments</td><td>Two non-null constant strings: catalog name and JSON envelope. + Duplicate members, trailing JSON and unknown options are rejected before catalog IO.</td></tr> + <tr><td>Targets</td><td>One table on <code>main</code>, one ordered index field group. + Latest snapshot by default; explicit selector: + <code>{"kind":"snapshot","snapshot_id":"123","tag":null}</code>.</td></tr> + <tr><td>DSL</td><td><code>match_all</code>, <code>match_none</code>, scalar <code>term</code>, + <code>terms</code>, <code>range</code>, <code>exists</code>, <code>match</code>, + <code>match_phrase</code>, <code>bool</code> and root-level <code>knn</code> with an optional + filter. The persisted field mappings and analyzers determine query semantics. Java numeric terms are typed, integer fields require integral values, and date bounds use epoch milliseconds.</td></tr> Review Comment: [Med] documented date unit it incorrect and can silently return wrong results:: This unit is not correct for every temporal type. `ESIndexGlobalIndexWriter.extractScalar` stores `DATE` as epoch days, `TIME` as milliseconds-of-day, and only `TIMESTAMP` / `TIMESTAMP_LTZ` as epoch milliseconds; the query parser passes these numeric bounds directly to the Lucene `LONG` field. A client following this sentence and sending epoch milliseconds for a `DATE` column will therefore silently match the wrong range (usually nothing). Could we document the unit per logical type and update the parser's `"dates use epoch millis"` validation message accordingly? A focused DATE/TIME/TIMESTAMP test would also protect the contract. ########## paimon-elasticsearch/pom.xml: ########## @@ -28,8 +28,8 @@ under the License. <version>2.2-SNAPSHOT</version> </parent> - <artifactId>paimon-eslib</artifactId> - <name>Paimon : ESLib Index</name> + <artifactId>paimon-elasticsearch</artifactId> Review Comment: [High] Renaming the Maven artifact/profile breaks existing users. Renaming the directory is harmless, but changing the published artifact from `org.apache.paimon:paimon-eslib` to `org.apache.paimon:paimon-elasticsearch` (and removing the paimon-eslib profile) makes existing downstream builds and plugin deployment scripts fail immediately on upgrade. Keeping the Java packages and archive format compatible does not help those consumers because Maven can no longer resolve the old coordinate. Could we retain `paimon-eslib` as a deprecated relocation/shim artifact and keep the old profile as an alias for at least one release? Maven's relocation metadata or a small compatibility module depending on `paimon-elasticsearch` would preserve the upgrade path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
