Kamal, Thanks for your time on reading the KIP and great questions. Here is the reply to some of the questions:
KC1: For transactional message, we plan to support it in the second phase of this feature. Transactional message support will introduce some complexities on the consumer side, the consumer will need to download both data log segment file as well as transactional index file and producer snapshot file from tiered storage and then filter out aborted transaction messages based transactional index file and producer snapshot, those code snippet does exist on the server side. We would need to do quite a bit of code refactoring to make them usable on the consumer side; KC2: For access to S3/GCS/Azure, the consumer might need extra certificates or they might just rely on configuring the right AWS IAM role/policy to access S3 (so that part of the configuration is actually not present in Kafka’s properties file). This is very similar to the configuration of RemoteStorageManager in KIP-405, we will address them in the KIP for Java consumer implementation. KC3: For queuing support, does share groups/topics currently available in tiered storage? I don’t seem to seeing queue related code in KIP-405 code paths. Again the queue support might needs to be addressed in the next phase of this feature. To support queues, we would need to replicate the redirecting mechanism we designed for FetchRequest/FetchResponse to ShareFetchRequest/ShareFetchResponse, this part should be easy to implement. But for KIP-932, the share partition leader needs to do the bookkeeping work for the state of the messages between SPSO and SPEO (the in-flight messages), the share partition leader is currently living on the broker, in order for this still to work for the consumer who wants to bypass broker, we would have to maintain 2 channels. The consumer can download the bulk message data directly from tiered storage, but the consumer still needs to work with the share-partition-leader-broker to exchange the state information of the in-flight message; On 2025/12/02 11:04:13 Kamal Chandraprakash wrote: > Hi Haiying, > > Thanks for the KIP! > > 1. Do you plan to add support for transactional consumers? Currently, the > consumer doesn't return the aborted transaction records to the handler. > 2. To access the remote storage directly, the client might need additional > certificates / keys. How do you plan to expose those configs on the client? > 3. Will it support the Queues for Kafka feature KIP-932 > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka>? > And so on. > > -- > Kamal > > On Tue, Dec 2, 2025 at 10:29 AM Haiying Cai via dev <[email protected]> > wrote: > > > For some reason, the KIP link was truncated in the original email. Here > > is the link again: > > > > KIP: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1248%3A+Allow+consumer+to+fetch+from+remote+tiered+storage > > > > Henry Haiying Cai > > > > On 2025/12/02 04:34:39 Henry Haiying Cai via dev wrote: > > > > > > Hi all, > > > > > > > > > > > > > > > I would like to start discussion on KIP-1248: Allow consumer to fetch > > from remote tiered storage > > > > > > > > > > > > KIP link: KIP-1248: Allow consumer to fetch from remote tiered storage - > > Apache Kafka - Apache Software Foundation > > > > > > | > > > | > > > | | > > > KIP-1248: Allow consumer to fetch from remote tiered storage - Apache > > Ka... > > > > > > > > > | > > > > > > | > > > > > > | > > > > > > > > > > > > > > > The KIP proposes to allow consumer clients to fetch from remote tiered > > storage directly to avoid hitting broker's network capacity and cache > > performance. This is very useful to serve large backfill requests from a > > new or fallen-off consumer. > > > > > > > > > > > > > > > Any feedback is appreciated. > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Henry >
