Also just to add I did not add this in the first place as I think Avro itself does not have the Nanosecond logical type.
On Wed, Apr 16, 2025 at 7:01 PM pritam kumar <kumarpritamm...@gmail.com> wrote: > Thanks Sudesh, for taking a look at this. I am already working on > extending this for nanosecond precision as most sinks like iceberg have > started giving nanosecond precision options. > > On Wed, Apr 16, 2025 at 4:41 PM Sudesh Wasnik <wasnik...@gmail.com> wrote: > >> Hi Pritam ! Thanks for the KIP ! >> Let’s extend the KIP to also add support for Nanosecond precision! >> >> Thanks >> Sudesh >> >> On 2025/04/05 01:30:49 pritam kumar wrote: >> > Hi Kafka Community, >> > Sorry due to some changes I had to change the link to the kip. >> > Here is the updated KIP link: >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1154%3A+Extending+support+for+Microsecond+Precision+for+Kafka+Connect >> > >> > On Sat, Apr 5, 2025 at 12:14 AM pritam kumar <ku...@gmail.com> >> > wrote: >> > >> > > Hi Kafka Community, >> > > >> > > I’d like to start a discussion on KIP-1153: Extending Support for >> > > Microsecond Precision for Kafka Connect >> > > < >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect >> > >> > > . >> > > >> > > The primary motivation behind this KIP is to enhance the precision of >> > > timestamp handling in Kafka Connect. Currently, Kafka Connect is >> limited to >> > > millisecond-level precision for timestamps. However, many modern data >> > > formats and platforms have moved beyond this limitation: >> > > >> > > - >> > > >> > > Formats such as *Avro*, *Parquet*, and *ORC* support microsecond (and >> > > even nanosecond) precision. For example, Avro specifies support for >> > > timestamp-micros (spec link >> > > <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>). >> > > - >> > > >> > > Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi* >> > > offer *microsecond and nanosecond precision* for time-based fields, >> > > making millisecond precision inadequate for accurate data replication >> and >> > > analytics in some use cases. >> > > >> > > This gap can lead to *loss of precision* when transferring data >> through >> > > Kafka Connect, especially when interacting with systems that depend on >> > > high-resolution timestamps for change tracking, event ordering, or >> > > deduplication. >> > > >> > > The goal of this KIP is to: >> > > >> > > - >> > > >> > > Introduce microsecond-level timestamp handling in Kafka Connect schema >> > > and data representation. >> > > - >> > > >> > > Ensure connectors (both source and sink) can leverage this precision >> > > when supported by the underlying data systems. >> > > - >> > > >> > > Maintain backward compatibility with existing millisecond-based >> > > configurations and data. >> > > >> > > We welcome community feedback on: >> > > >> > > - >> > > >> > > Potential implementation concerns or edge cases we should address >> > > - >> > > >> > > Suggestions for schema enhancements or conversion strategies >> > > - >> > > >> > > Impacts on connector compatibility and testing >> > > >> > > Looking forward to your thoughts and input on this proposal! >> > > >> > > Thanks. >> > > Link to the KIP. >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect >> > > < >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API >> > >> > > >> > > >> >> Sent with a Spark >> >
