Ujjawal Kumar created HBASE-30366:
-------------------------------------
Summary: Support for custom metadata in HFiles
Key: HBASE-30366
URL: https://issues.apache.org/jira/browse/HBASE-30366
Project: HBase
Issue Type: Improvement
Components: Coprocessors, HFile
Reporter: Ujjawal Kumar
HFiles today metadata in its file format at couple of place - FileInfo, trailer
and metadata blocks.
Supporting custom metadata in Hfile (via coprocessor hooks) can allow users to
inject more rich metadata. One example of this is capturing tenant level stats
in phoenix (custom scanner reading the data row by row and capturing the stats
which is written to hfile during writes)
The place to put this metadata can be either FileInfo or metadata blocks.
However we would want to read this lazily only when the reader explicitly asks
for it.
Both trailer and fileinfo are always read during HFile open which also would
need the metadata to be small
With Metadata block, we can support both lazy read and large metadata.
This can be done via something like
1. Add the restricted coprocessor API `StoreFileMetadataSink` with
`appendMetaBlock(String, Writable)` and, if useful for other callers,
`appendFileInfo(byte[], byte[])`.
2. Add flush and compaction pre-close observer hooks that receive the final
operation scanner and a `StoreFileMetadataSink` bound to the still-open output
writer.
3. Dispatch the flush hook after `performFlush(...)` succeeds and before
`DefaultStoreFlusher.finalizeWriter(...)` closes the file.
4. Dispatch the compaction hook after `performCompaction(...)` succeeds and
before `Compactor.commitWriter(...)` closes/commits the file.
5. Add generic `appendMetaBlock(...)` delegation from `StoreFileWriter` through
`SingleStoreFileWriter` to `HFile.Writer.appendMetaBlock(...)`; do not expose
the complete writer to the coprocessor.
6. Ensure hook or metadata-write failures abort the output instead of
committing an HFile without the requested metadata.
7. Define invocation semantics for every physical output, including
live/historical writers and rolled multi-output compactions.
8. Add HFile v4 - multi-tenant support
([HBASE-29588|https://issues.apache.org/jira/browse/HBASE-29588) section-close
metadata injection and section-aware metadata lookup. A file-level close hook
is too late for previously closed v4 sections, and the current name-only lookup
returns only one matching section block.
HFile v2/v3 already support named global META blocks, so no v2/v3 on-disk
format change is required.
For V4, this metadata block is maintained at section level
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
