abernardi597 opened a new issue, #16392:
URL: https://github.com/apache/lucene/issues/16392
### Description
Lucene readers know a great deal about how bytes are laid out on disk —
which byte
ranges of a segment file belong to which field — but that information is not
observable
today. There is no general way to ask "how many bytes of file X does field Y
occupy?"
across postings, doc values, KNN vectors, points, etc.
This makes it hard to answer operationally important questions:
- Which fields dominate a segment's on-disk footprint?
- Given a memory-residency signal (e.g. `mincore(2)`/`cachestat(2)`), which
fields are
"cold" (least resident in the page cache) and therefore candidates for
cost or latency
attention?
- How is a field's storage split across the formats that store it (postings
vs. doc
values vs. points)?
Today the only ways to get at this are indirect and fragile: summing
whole-file sizes
(no per-field breakdown), or reflecting into codec internals to recover
per-field file
pointers (brittle, breaks on every codec change).
## Precedent
`KnnVectorsReader#getOffHeapByteSize(FieldInfo)` (#14426) already
established that a
reader can report its per-field on-disk size, and that this is useful enough
to live on
the reader API. That method returns a per-field, per-extension **total**.
This proposal
generalizes the same idea in two ways:
1. To **any** format (not just KNN vectors)
2. From a per-field *total* to a per-field *byte-range layout* so a consumer
can
correlate ranges with offset-addressed signals like page-cache residency,
read-access histograms, etc.
## Proposed shape
### The Data: an `Extent` and its `Attribution`
A contiguous byte range, and what it belongs to:
```java
/** A contiguous byte range within a file, and a descriptor of its purpose.
*/
record Extent(long offset, long length, Attribution attribution) {}
/** What the bytes in an Extent belong to. */
sealed interface Attribution {
/** Bytes belonging to a single field. */
record Field(String name) implements Attribution {}
/** Codec-declared overhead: header, footer, padding, or other
field-independent metadata. */
record Structural() implements Attribution {}
/** Bytes no reader described: a gap within a file, or a file no reader
reports. */
record Unclaimed() implements Attribution {}
}
```
Modeling attribution as a sealed type (rather than a nullable field name or
a set)
distinguishes the cases that consumers genuinely need to treat differently
(attributed,
structural overhead, and not-yet-described) while leaving room to add other
kinds
later (e.g. a range shared by several fields) without silently breaking
existing consumers, who
get a compile error until they handle the new case.
### The API
```java
public interface DiskAccountable {
/**
* On-disk layout of the files this component manages, keyed by file name.
Each value is
* a list of non-overlapping Extents sorted by offset that together cover
the whole file
* [0, fileLength): ranges no reader describes appear as
Attribution.Unclaimed rather than
* being omitted, so every byte is accounted for exactly once. Returns an
empty map by default.
*/
default Map<String, List<Extent>> getDiskLayout() throws IOException {
return Map.of();
}
}
```
The natural home is per-segment, and the file key is built up as the layout
bubbles up
through the reader tiers — each tier contributing the part of the name it
knows:
1. **Leaf codec readers** (`FieldsProducer`, `DocValuesProducer`,
`KnnVectorsReader`,
`PointsReader`, ...) describe the files they manage. A reader has its
extension and its
`segmentSuffix` (from `SegmentReadState`) but not the segment name, so it
can name a
file by everything *except* that prefix — e.g. a flat vectors reader's
`.vec`.
2. **Per-field formats** (`PerFieldKnnVectorsFormat`, etc.) delegate to one
inner reader
per format+instance, each constructed with a distinguishing
`segmentSuffix`
(`Lucene99_0`, `Lucene99_1`, ...). That suffix is what separates the
multiple physical
files a per-field format produces — `_Lucene99_0.vec` vs
`_Lucene99_1.vec`.
3. **`CodecReader`** merges its child readers' layouts, and
**`SegmentReader`** — the
only tier that knows the segment name and holds the segment's file list —
completes
each key into a full file name (`_5_Lucene99_0.vec`) and reconciles the
merged result
against `SegmentCommitInfo#files()`: any file no child reader described
is added as a
whole-file `Unclaimed` extent. The segment-level result is therefore
keyed by
globally-unique full file names and covers every file in the segment —
including
formats that do not (yet) implement `DiskAccountable`, which simply
surface as
`Unclaimed` and can be attributed incrementally as readers adopt the
interface.
A consumer walks a `SegmentReader`'s layout, correlates each `Extent` with
whatever
offset-addressed signal it cares about, and aggregates per `Attribution`.
## Open questions for discussion
1. **Cost.** Building a per-file `Extent` graph for every field of every
segment on every
call has a GC/CPU cost that grows with field count (which can be high).
Should the base
API lean toward a visitor/callback style (accumulate into a
caller-supplied sink,
allocating nothing) rather than materializing an actual `Map<String,
List<Extent>>`? A segment's
layout should not change after it is written, so caching by segment
**core** identity
(`LeafReader#getCoreCacheHelper`, which is stable across new deletions)
is the intended
mitigation, but the base API shape still matters. Interested in
preferences here.
2. **Attribution kinds.** `Field` / `Structural` / `Unclaimed` covers the
basics. Is
there appetite for modeling a range shared by multiple fields (e.g.
deduplicated
vectors), or is that better left out until a concrete need exists?
Sharing also breaks
the one-field-one-contiguous-run intuition — a shared range belongs
partly to several
fields at once — so any per-field total over shared ranges needs an
explicit split
policy (equal? by reference count?) rather than a simple sum.
3. **Which readers.** A useful first cut covers postings, doc values, KNN
vectors, and
points (BKD). Stored fields and term vectors are block-compressed with
interleaved
fields and are much harder to attribute per-field; they would surface as
`Unclaimed`
until/unless someone tackles them.
4. **Full coverage vs. best-effort.** This sketch proposes the segment-level
result cover
`[0, fileLength)` for every file (gaps and unreported files become
`Unclaimed`), which
makes totals reconcile against the directory's file sizes. The
alternative — allow
gaps, omit unreported files — is cheaper but leaves the caller to
reconcile. Preference?
Reconciliation is a diff against `SegmentCommitInfo#files()` in both
directions: a file
in `files()` that no reader described becomes a whole-file `Unclaimed`
(the common
case). The reverse — a reader reporting a file *not* in `files()` — would
indicate the
reader's view and the commit's file list disagree; the sketch treats that
as a bug to
surface (fail fast) rather than silently drop, but the API contract for
it is worth
pinning down.
5. **Keying.** Lucene already models a file name as `(segmentName,
segmentSuffix, ext)`
and constructs it with `IndexFileNames#segmentFileName(...)`; a reader
has its
`segmentSuffix` and `ext` at construction (via `SegmentReadState`) but
not the segment
name. So rather than have readers emit an opaque suffix string that a
higher tier
concatenates, the layout key could align with that existing model —
readers key by
`segmentSuffix`/`ext`, and `SegmentReader` completes the name with the
canonical
`segmentFileName(...)` (the same call the write path used, so it
round-trips against
`files()` exactly). This avoids re-implementing name construction and its
edge cases.
Worth deciding what the key type should be (structured vs. string) at the
interface
level.
6. **Compound files.** Under `CompoundFormat`, child readers describe
*logical* files
(`.doc`, `.dvd`, ...) at offsets within those logical files, but on disk
there is only
the physical `.cfs` (plus `.cfe`). A raw merge would therefore key by
logical files
that do not physically exist. `SegmentReader` is the tier that could
reconcile this: it
knows whether the segment is compound and holds the compound `Directory`,
which maps
each logical file to its offset within `.cfs`. Options range from
translating logical
extents into `.cfs`-relative ones (preserves whole-byte accounting, but
needs the
compound directory to expose per-entry offsets), to reporting the `.cfs`
as a single
`Unclaimed` file (no per-field breakdown inside compound segments), to
leaving CFS out
of scope initially. This likely needs input from folks who own
`CompoundFormat`.
## Prior art / relationship to existing APIs
- `getOffHeapByteSize` (#14426): per-field, per-extension totals for KNN
vectors. This is
the range-level generalization, across all formats.
- `Accountable`/`ramBytesUsed`: RAM, not disk; whole-component, not
per-field.
- `CheckIndex` / `SegmentInfos#files()`: whole-file sizes, no per-field
breakdown.
Happy to put up a draft PR implementing this across the readers above plus
the
`CodecReader`/`SegmentReader` composition, if there's interest in the
direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
