razgaou opened a new pull request, #784:
URL: https://github.com/apache/atlas/pull/784
## What changes were proposed in this pull request?
This PR adds semantic search to Apache Atlas backed by OpenSearch k-NN
embeddings, with a standalone semantic indexer (Kafka consumer), repair tool,
REST/UI integration
Highlights:
* Semantic opt-in: Semantic search is off by default until operators set
atlas.semantic.enabled=true on Atlas and the semantic indexer/repair tool; that
only works if the OpenSearch vertex index was created with
`atlas.graph.index.search.opensearch.create.ext.knn=true` (so index.knn is set
at index creation, it cannot be turned on later via config alone on an existing
index).
* Standalone indexer: Kafka-driven re-embedding; health endpoint;
cooperative shutdown and backend retry when OpenSearch/ML/graph are temporarily
unavailable.
* Repair tool: OpenSearch PIT (Point In Time) scan to backfill missing
embeddings (or --all / --guid)
* Two main new REST API: `api/atlas/v2/search/semantic`
`api/atlas/v2/search/similar` , and indexer health `/health `
## How was this patch tested?
* Unit / module tests:
mvn -pl
repository,services/atlas-semantic-indexer,tools/atlas-semantic-repair,graphdb/janusgraph-opensearch
-am test
Result: BUILD SUCCESS â 95 tests, 0 failures, 0 errors, 0 skipped (~60s).
* Sample data: quick_start.py inside the Atlas container, All entities were
indexed with the vector embedding, semantic search and similar features are
working as expected from Atlas UI.
* Scale/Load: ~610k bulk entities with descriptions and classifications..,
on local Docker env (Spec: OpenSearch: 2 GB RAM, indexer service: 512 MB RAM,
Docker Desktop was capped at ~10 CPUs ):
* Indexer: ~170k: Across the Kafka catch-up run, embedder log lines
reported failed=0 on essentially every batch (~174k indexed in those lines).
Average throughput for the run was about 63 embeddings/s (~45 min from indexer
start to the 170k stop).
* repair tool: After clearing embeddings, repair (batch 500, missing
mode) re-indexed to ~173.5k with failed=0 on repair progress logs in ~43 min
(~67/s average).â
* API: Semantic and similar REST returned SEMANTIC results with expected
hit counts (topK 5â10); no 503/500 during checks at ~170kâ173k indexed entities.
* More details:
https://github.com/razgaou/atlas/blob/ATLAS-5416_semantic_search/dev-support/atlas-docker/SemanticSearch.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
