[jira] [Commented] (KAFKA-3846) Connect record types should include timestamps
[ https://issues.apache.org/jira/browse/KAFKA-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435918#comment-15435918 ] Ewen Cheslack-Postava commented on KAFKA-3846: -- [~michael.andre.pearce] Since the timestamp wasn't available until recently, the existing connector has to use wall-clock time. I imagine it'd be a simple patch to use the timestamp, if available, from the record instead. > Connect record types should include timestamps > -- > > Key: KAFKA-3846 > URL: https://issues.apache.org/jira/browse/KAFKA-3846 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.0.0 >Reporter: Ewen Cheslack-Postava >Assignee: Shikhar Bhushan > Labels: needs-kip > Fix For: 0.10.1.0 > > > Timestamps were added to records in the previous release, however this does > not get propagated automatically to Connect because it uses custom wrappers > to add fields and rename some for clarity. > The addition of timestamps should be trivial, but can be really useful (e.g. > in sink connectors that would like to include timestamp info if available but > when it is not stored in the value). > This is public API so it will need a KIP despite being very uncontentious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3846) Connect record types should include timestamps
[ https://issues.apache.org/jira/browse/KAFKA-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435842#comment-15435842 ] Michael Andre Pearce (IG) commented on KAFKA-3846: -- This looks great, is any documents updated to show how we configure bits like TimebasedPartitioner mentioned here http://docs.confluent.io/3.0.0/connect/connect-hdfs/docs/configuration_options.html to use this? Also is it possible to store this timestamp ( and other meta) in the hdfs/hive record it seems we can only store the payload. From the config options in connect. > Connect record types should include timestamps > -- > > Key: KAFKA-3846 > URL: https://issues.apache.org/jira/browse/KAFKA-3846 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.0.0 >Reporter: Ewen Cheslack-Postava >Assignee: Shikhar Bhushan > Labels: needs-kip > Fix For: 0.10.1.0 > > > Timestamps were added to records in the previous release, however this does > not get propagated automatically to Connect because it uses custom wrappers > to add fields and rename some for clarity. > The addition of timestamps should be trivial, but can be really useful (e.g. > in sink connectors that would like to include timestamp info if available but > when it is not stored in the value). > This is public API so it will need a KIP despite being very uncontentious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3846) Connect record types should include timestamps
[ https://issues.apache.org/jira/browse/KAFKA-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357843#comment-15357843 ] ASF GitHub Bot commented on KAFKA-3846: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1537 > Connect record types should include timestamps > -- > > Key: KAFKA-3846 > URL: https://issues.apache.org/jira/browse/KAFKA-3846 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.0.0 >Reporter: Ewen Cheslack-Postava >Assignee: Shikhar Bhushan > Labels: needs-kip > Fix For: 0.10.1.0 > > > Timestamps were added to records in the previous release, however this does > not get propagated automatically to Connect because it uses custom wrappers > to add fields and rename some for clarity. > The addition of timestamps should be trivial, but can be really useful (e.g. > in sink connectors that would like to include timestamp info if available but > when it is not stored in the value). > This is public API so it will need a KIP despite being very uncontentious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3846) Connect record types should include timestamps
[ https://issues.apache.org/jira/browse/KAFKA-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15346891#comment-15346891 ] Shikhar Bhushan commented on KAFKA-3846: https://cwiki.apache.org/confluence/display/KAFKA/KIP-65%3A+Expose+timestamps+to+Connect > Connect record types should include timestamps > -- > > Key: KAFKA-3846 > URL: https://issues.apache.org/jira/browse/KAFKA-3846 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.0.0 >Reporter: Ewen Cheslack-Postava >Assignee: Shikhar Bhushan > Labels: needs-kip > Fix For: 0.10.1.0 > > > Timestamps were added to records in the previous release, however this does > not get propagated automatically to Connect because it uses custom wrappers > to add fields and rename some for clarity. > The addition of timestamps should be trivial, but can be really useful (e.g. > in sink connectors that would like to include timestamp info if available but > when it is not stored in the value). > This is public API so it will need a KIP despite being very uncontentious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3846) Connect record types should include timestamps
[ https://issues.apache.org/jira/browse/KAFKA-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343117#comment-15343117 ] ASF GitHub Bot commented on KAFKA-3846: --- GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/1537 KAFKA-3846: include timestamp in Connect record types KIP to come You can merge this pull request into a Git repository by running: $ git pull https://github.com/shikhar/kafka kafka-3846 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1537.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1537 commit 2dc7e91c1ec06fcf987dda07a63acf45e1b2d13e Author: Shikhar BhushanDate: 2016-06-22T00:42:48Z KAFKA-3846: include timestamp in Connect record types; add Builder for `SourceRecord` `SinkRecord` gets `timestampType` and `timestamp` `SourceRecord` gets `timestamp` `SourceRecord.Builder` is the new preferred way to construct `SourceRecord`s > Connect record types should include timestamps > -- > > Key: KAFKA-3846 > URL: https://issues.apache.org/jira/browse/KAFKA-3846 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.0.0 >Reporter: Ewen Cheslack-Postava >Assignee: Shikhar Bhushan > Labels: needs-kip > Fix For: 0.10.1.0 > > > Timestamps were added to records in the previous release, however this does > not get propagated automatically to Connect because it uses custom wrappers > to add fields and rename some for clarity. > The addition of timestamps should be trivial, but can be really useful (e.g. > in sink connectors that would like to include timestamp info if available but > when it is not stored in the value). > This is public API so it will need a KIP despite being very uncontentious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)