bamaer opened a new issue, #8857:
URL: https://github.com/apache/hop/issues/8857
### What needs to happen?
### What
Apache Hop now has every building block for GraphRAG: Text chunker, Embed
text, Extract graph, Graph output with vector properties, the Graph index
action with vector indexes, Graph vector search, Graph query and Language Model
Chat. What's missing is a worked example that puts them together, tests that
keep the combination working across graph databases, and a how-to.
This issue contributes those, adapted from a GraphRAG project that runs on
Memgraph, Neo4j, FalkorDB and Apache AGE + pgvector.
### Contents
**Graph model** (`neo4j-graph-model`): `Document -HAS_CHUNK-> Chunk
-MENTIONS-> Entity -RELATED_TO-> Entity`.
- Chunks carry their text and embedding.
- Entities merge on a normalized key (the name in lowercase, without spaces
or punctuation), so "Hop Run" and "hop-run" are one node.
- RELATED_TO carries the relationship type and the chunk it was read from.
**Samples** (`plugins/tech/ai/src/main/samples`):
- `graphrag-index.hpl` / `.hwf`: chunk, embed and write documents; then
Extract graph per chunk and write entities, mentions and relationships. The
workflow creates the lookup and vector indexes first.
- `graphrag-retrieve-graph.hpl` / `graphrag-retrieve-hybrid.hpl`, which
return one row per chunk:
- vector search (Graph vector search, or pgvector for AGE);
- graph expansion: chunks connected through the entities of the top
matches and of the question, skipping hub entities;
- per chunk, the facts that relate to the question;
- the relationships of the entities the question names.
- `graphrag-ask.hpl`: embed the question, retrieve, and answer with Language
Model Chat.
- `graphrag-refresh.hwf`: remove what changed documents produced (chunks,
and relationships by source chunk), extract again, remove orphaned entities.
- Hybrid variant for Apache AGE: AGE has no vector index, so embeddings stay
in pgvector and the graph holds the same chunk ids. This is the fully
Apache-licensed setup.
**Integration tests**, in the memgraph, neo4j, falkordb and age suites. No
language model is needed: fixed chunks with 3-dimensional embeddings, and fixed
Extract graph output rows.
- graph contents: node and relationship counts, and relationship provenance;
- retrieval: chunk order, a chunk the graph adds, the facts and the question
facts;
- refresh: removing a document leaves no stale chunks, relationships or
entities.
**How-to** in the user manual: choosing a database, setup, indexing, asking,
refreshing, and the known limits.
### Prerequisites and related issues
- #8836 (variables in Graph index and Graph constraint names): needed to
name the vector index from a variable, so tests and real indexes can share a
database. Until then the samples use a fixed index name.
- #8835 (Thinking option, merged): extraction with thinking models is
several times slower. The samples set Thinking to OFF.
- Graph output merges a relationship on its two nodes only (`MERGE
(a)-[r:TYPE]->(b)`, then `SET` the properties). A second relationship between
the same nodes overwrites the first, so the samples write RELATED_TO with Graph
query (`MERGE (s)-[:RELATED_TO {type: $type, chunk_id: $chunk_id}]->(t)`). A
separate issue could let Graph output use relationship properties as merge keys.
### Notes
- Extraction quality on 40 chunks of the Hop user manual: hosted Mistral
Small and local qwen3.5 (thinking off) keep 95–99% of entities grounded in the
chunk text. Some local instruct models invent entities, or copy examples from
the extraction instructions into the graph. The how-to should recommend
checking extraction quality on a sample first, and avoiding named examples in
the instructions.
- FalkorDB is supported by the samples and tests but is not the default: the
server is SSPL-licensed, and Hop ships only the client.
### Issue Priority
Priority: 2
### Issue Component
Component: Neo4j
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
