daniel added a comment.
Here's a rough outline of the pre-fetching infrastructure for labels and other
Terms:
We need a TermCache service like this:
```
TermCache {
/**
* Update terms for the given entity. Any old terms associated with the
entity are discarded.
**/
public function updateTerms( EntityId $entityId, Fingerprint $terms );
/**
* Loads a set of terms into memory, for later use by getTerms()
**/
public function prefetchTerms( EntityId $entityId, Fingerprint $terms );
/**
* Get terms of the given types for the given entities, in the given
languages (or all languages).
**/
public function getTerms( EntityId[] $entityIds, $termTypes, $languages );
}
```
This interface is somewhat similar to the TermIndex interface. We should
consider cleaning up TermIndex, and using that.
The TermCache makes use of a persistent cache (optionally shared between wikis)
aka memcached, and local in-process caching. It would be used as follows:
- ChangeHandler calls updateTerms() when it is notified of an Entity being
updated. If the cache is shared, the repo does this immediately when the terms
associated with an Entity are modified. In both cases, terms from the
Fingerprint are placed into memcached using setMulti(), one entry per term. The
cache key will contain the entity Id, term type, and language (plus possibly a
wiki id, version id, etc). Multi-value terms (aliases) are stored as a list of
values. A special key that does not include the type or language parts is used
to store a list of all the keys used for a given entity, to allow these keys to
be purged when updateTerms() is called again for the same entity.
- A hook like ChangesListInitRows is used to trigger prefetchTerms();
prefetchTerms() uses getMulti() to fetch all the desired terms from memcached,
and stores them locally in a hash. PROBLEM: if some terms are missing, we do
not know whether they are uncached, or do not exist. Negative caching and/or
checking the key list should be used. TBD: Decide whether TermCache should know
how to fetch uncached terms.
- A hook like LinkBegin may use a LabelLookup at is based upon a TermCache
and TermLookup to get terms associated with item pages; If the desired label
was previously fetched via prefetchTerms(), this should be very quick.
The main issue that remains to be decided is when and where cache misses are
resolved (by looking at the wb_terms table); one complication is that it's
unclear whether we should load all terms of an entity in such a case, or just
the one we currently need.
This architecture is intended to minimize i/o volume as well as round trips to
memcached. It still means putting a large number of small entries into
memcached with no expiration, possibly swamping it and pushing out high usage
entries with low usage labels. A "randomized put" strategy could be used to
mitigate this issue by writing only every Kth entry to the cache, giving
frequently used labels a higher chance of being cached.
TASK DETAIL
https://phabricator.wikimedia.org/T74309
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
<username>.
To: daniel
Cc: wikidata-bugs, Tobi_WMDE_SW, Liuxinyu970226, Wikidata-bugs, daniel
_______________________________________________
Wikidata-bugs mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
