IiCEight opened a new issue, #435:
URL: https://github.com/apache/hugegraph-ai/issues/435

   ### Search before asking
   
   - [x] I had searched in the 
[feature](https://github.com/apache/hugegraph-ai/issues?q=is%3Aissue+label%3A%22Feature%22)
 and found no similar feature requirement.
   
   
   ### Feature Description (功能描述)
   
   
   Related: [#344](https://github.com/apache/hugegraph-ai/issues/344) focuses 
on chunk-level extraction concurrency. This request addresses deduplication and 
entity resolution after chunk extraction.
   
   ## Feature Description (功能描述)
   
   Graph extraction processes chunks independently and concatenates their 
vertices and edges. Repeated entities can appear multiple times, while 
different names for the same entity can remain disconnected. There is no 
cross-chunk policy for resolving aliases or conflicting properties before 
import.
   
   Observed reproduction using DeepSeek through `POST /graph/extract`, with two 
input strings, `split_type: "document"`, and review disabled. The inline schema 
defines `person`, `company`, and `city` vertices keyed by `name`, plus 
`works_for` and `located_in` edges.
   
   ```text
   Chunk 1: Alice works for Acme.
   Chunk 2: Acme Ltd., also known as Acme, is based in Beijing.
   ```
   
   The request returned HTTP 200 in 8.31 seconds, with `status: "succeeded"`, 
no warnings, 4 vertices, and 2 edges:
   
   ```text
   1:Alice --works_for--> 2:Acme
   2:Acme Ltd. --located_in--> 3:Beijing
   ```
   
   The explicit alias remained two company identities, leaving the subgraphs 
disconnected. After resolving the supported alias, the expected graph has 3 
vertices and 2 edges:
   
   ```text
   Alice --works_for--> Acme --located_in--> Beijing
   ```
   
   This records one observed run; model output may vary.
   
   ## Proposal
   
   Add an optional internal stage after chunk extraction. Run deterministic 
checks first; use the LLM only as an optional judge for unresolved candidates:
   
   1. **Deterministic duplicate detection:** group vertices by 
schema-compatible labels and compare typed IDs and declared identity keys. 
Consolidate matching identities without LLM calls. Conflicting authoritative 
identifiers rule out a merge; conflicting properties are reported rather than 
overwritten.
   2. **Deterministic candidate matching:** for remaining vertices, use 
character n-grams/token overlap and a lexical similarity score, such as 
Jaro-Winkler, to shortlist potential aliases within compatible groups. Preserve 
punctuation-sensitive identifiers and keep short names eligible. Normalized 
names or similarity scores alone do not authorize a merge.
   3. **Optional LLM judgment:** only send unresolved candidate pairs to the 
existing configured model, including both extracted entities, the relevant 
schema, and their original source chunks. Request an internal `same`, 
`different`, or `uncertain` decision with supporting evidence. If judgment is 
disabled, fails, exceeds its budget, or remains uncertain, keep the entities 
separate. Bound candidate counts, model calls, and concurrency.
   4. **Apply confirmed merges:** combine deterministic matches and 
model-confirmed matches only after checking group-wide identity constraints and 
contradictory decisions. Resolve mappings to stable canonical IDs, rewrite edge 
endpoints, merge compatible properties with provenance, and validate the 
combined graph. Deduplicate relationships using schema identity while 
preserving legitimate parallel edges and self-loops.
   
   For the reproduction above, deterministic matching should shortlist `Acme` 
and `Acme Ltd.`. The optional judge can then use the original second chunk's 
"also known as Acme" statement to confirm the alias.
   
   
   ### Are you willing to submit a PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to