Why do you need to change the kafka native timestamp? It is the time the message was produced, defined by the configuration log.message.timestamp.type<https://kafka.apache.org/documentation/#brokerconfigs_log.message.timestamp.type>, by default the time the record was created client-side.
If you need to add a timestamp with other semantics then you should add another timestamp, either in the payload or as a header. Regards, ________________________________ From: Peter Fischer <pfisc...@wikimedia.org> Sent: Thursday, October 31, 2024 00:34 To: dev@spark.apache.org <dev@spark.apache.org> Subject: [Spark SQL] KafkaWriteTask: allow customising timestamp - PR You don't often get email from pfisc...@wikimedia.org. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> EXTERNAL SENDER. Do not click links or open attachments unless you recognize the sender and know the content is safe. DO NOT provide your username or password. Hi! We wrote a wrapper around the kafka writer to add client-side schema validation. In the process we noticed that there was no way to change a kafka record's timestamp when writing. So we extended spark-sql-kafka to support it and would love to hear your feedback. JIRA: https://issues.apache.org/jira/browse/SPARK-50160 GitHub: https://github.com/apache/spark/pull/48695 -- Peter Fischer (he/him) Senior Software Engineer, Search Platform Wikimedia Foundation CONFIDENTIALITY NOTICE: This email message (and any attachment) is intended only for the individual or entity to which it is addressed. The information in this email is confidential and may contain information that is legally privileged or exempt from disclosure under applicable law. If you are not the intended recipient, you are strictly prohibited from reading, using, publishing or disseminating such information and upon receipt, must permanently delete the original and destroy any copies. We take steps to protect against viruses and other defects but advise you to carry out your own checks and precautions as Kambi does not accept any liability for any which remain. Thank you for your co-operation.
