GitHub user sijie added a comment to the discussion: Pulsar SQL: support user defined indexes
@KannarFr The indexes can be built in a background process using the approach that was used for compaction. The "compacted" ledger is essentially an "index" to the original data. The index maintains some forms of mapping between "keys" to the "offsets" to the original data. The "offset" is essentially the message-id which is referencing a ledger and an entry id. It doesn't matter if a ledger is in the bookkeeper or already offloaded to the tiered storage. GitHub link: https://github.com/apache/pulsar/discussions/18763#discussioncomment-4321905 ---- This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
