[
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-4144:
-----------------------------------
Description:
At the moment the timestamp extractor is configured via a {{StreamConfig}}
value to {{KafkaStreams}}. That means you can only have a single timestamp
extractor per app, even though you may be joining multiple streams/tables that
require different timestamp extraction methods.
You should be able to specify a timestamp extractor via
{{KStreamBuilder.stream()/table()}}, just like you can specify key and value
serdes that override the StreamConfig defaults.
KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
Specifying a per-stream extractor should only be possible for sources, but not
for intermediate topics. For PAPI we cannot enforce this, but for DSL
{{through()}} should not allow to set a custom extractor by the user. In
contrast, with regard to KAFKA-4785, is must internally set an extractor that
returns the record's metadata timestamp in order to overwrite the global
extractor from {{StreamsConfig}} (ie, set {{FailOnInvalidTimestampExtractor}}).
This change should be done in KAFKA-4785 though.
was:
At the moment the timestamp extractor is configured via a {{StreamConfig}}
value to {{KafkaStreams}}. That means you can only have a single timestamp
extractor per app, even though you may be joining multiple streams/tables that
require different timestamp extraction methods.
You should be able to specify a timestamp extractor via
{{KStreamBuilder.stream()/table()}}, just like you can specify key and value
serdes that override the StreamConfig defaults.
Specifying a per-stream extractor should only be possible for sources, but not
for intermediate topics. For PAPI we cannot enforce this, but for DSL
{{through()}} should not allow to set a custom extractor by the user. In
contrast, with regard to KAFKA-4785, is must internally set an extractor that
returns the record's metadata timestamp in order to overwrite the global
extractor from {{StreamsConfig}} (ie, set {{FailOnInvalidTimestampExtractor}}).
This change should be done in KAFKA-4785 though.
> Allow per stream/table timestamp extractor
> ------------------------------------------
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 0.10.0.1
> Reporter: Elias Levy
> Assignee: Jeyhun Karimov
> Labels: api, kip
>
> At the moment the timestamp extractor is configured via a {{StreamConfig}}
> value to {{KafkaStreams}}. That means you can only have a single timestamp
> extractor per app, even though you may be joining multiple streams/tables
> that require different timestamp extraction methods.
> You should be able to specify a timestamp extractor via
> {{KStreamBuilder.stream()/table()}}, just like you can specify key and value
> serdes that override the StreamConfig defaults.
> KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
> Specifying a per-stream extractor should only be possible for sources, but
> not for intermediate topics. For PAPI we cannot enforce this, but for DSL
> {{through()}} should not allow to set a custom extractor by the user. In
> contrast, with regard to KAFKA-4785, is must internally set an extractor that
> returns the record's metadata timestamp in order to overwrite the global
> extractor from {{StreamsConfig}} (ie, set
> {{FailOnInvalidTimestampExtractor}}). This change should be done in
> KAFKA-4785 though.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
