Hi Thanks for this KIP. I have a couple of comments
chia_00: the description "Workers running the old version will not recognize schemas with the new logical type names and will treat them as raw INT64, potentially causing data misinterpretation." seems to be wrong since the old worker will throw a NoClassDefFoundError, right? chia_01: does this KIP handle timestamps before 1970? The test plan mentions negative values, but the KIP does not define how fromLogical/toLogical convert them. chia_02: I guess an old sink can use TimestampConverter to convert the new types back to Timestamp to avoid compatibility issues? If so, it would be nice to document that in the migration path. Best, Chia-Ping On 2025/04/04 18:44:52 pritam kumar wrote: > Hi Kafka Community, > > I’d like to start a discussion on KIP-1153: Extending Support for > Microsecond Precision for Kafka Connect > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect> > . > > The primary motivation behind this KIP is to enhance the precision of > timestamp handling in Kafka Connect. Currently, Kafka Connect is limited to > millisecond-level precision for timestamps. However, many modern data > formats and platforms have moved beyond this limitation: > > - > > Formats such as *Avro*, *Parquet*, and *ORC* support microsecond (and > even nanosecond) precision. For example, Avro specifies support for > timestamp-micros (spec link > <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>). > - > > Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi* > offer *microsecond and nanosecond precision* for time-based fields, > making millisecond precision inadequate for accurate data replication and > analytics in some use cases. > > This gap can lead to *loss of precision* when transferring data through > Kafka Connect, especially when interacting with systems that depend on > high-resolution timestamps for change tracking, event ordering, or > deduplication. > > The goal of this KIP is to: > > - > > Introduce microsecond-level timestamp handling in Kafka Connect schema > and data representation. > - > > Ensure connectors (both source and sink) can leverage this precision > when supported by the underlying data systems. > - > > Maintain backward compatibility with existing millisecond-based > configurations and data. > > We welcome community feedback on: > > - > > Potential implementation concerns or edge cases we should address > - > > Suggestions for schema enhancements or conversion strategies > - > > Impacts on connector compatibility and testing > > Looking forward to your thoughts and input on this proposal! > > Thanks. > Link to the KIP. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API> >
