[
https://issues.apache.org/jira/browse/CALCITE-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16858187#comment-16858187
]
Matt Wang commented on CALCITE-3073:
------------------------------------
[~mingmxu] Using {color:#707070}*MSG_TIMESTAMP*{color} filed I think it's a
better idea, when user use KafkaAdapter, they know these field is already
defined:
{quote}| MSG_PARTITION | MSG_TIMESTAMP | MSG_OFFSET | MSG_KEY_BYTES |
MSG_VALUE_BYTES
{quote}
> Support to consume from timestamp in KafkaAdapter
> -------------------------------------------------
>
> Key: CALCITE-3073
> URL: https://issues.apache.org/jira/browse/CALCITE-3073
> Project: Calcite
> Issue Type: Improvement
> Components: kafka-adapter
> Reporter: Xu Mingmin
> Assignee: Matt Wang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Currently the KafkaAdapter consumes data from default
> offset(latest/earliest/last_offset) and runs forever.
> In other words, if the app runs at first time and user want to consume the
> past data, user must set the value of
> '{color:#707070}*auto.offset.reset'*{color} parameter to *earliest*.
> {quote}{{auto.offset.reset：What to do when there is no initial offset in
> Kafka or if the current offset does not exist any more on the server (e.g.
> because that data has been deleted):}}
> * earliest: automatically reset the offset to the earliest offset
> * latest: automatically reset the offset to the latest offset
> * none: throw exception to the consumer if no previous offset is found for
> the consumer's group
> * anything else: throw exception to the consumer.{quote}
> for example, suppose data in Kafka is retained for 7 days and you just want
> to read from the data of yesterday, if you could not control the start
> timestamp, you can only read from the earliest offset, it's very inefficient.
> If supporting to consume from special timestamp in KafkaAdapter will be a
> good idea for some cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
