[jira] [Commented] (KAFKA-7157) Connect TimestampConverter SMT doesn't handle null values

[Jeff Beagley (JIRA)]

    [ 
https://issues.apache.org/jira/browse/KAFKA-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16788436#comment-16788436
 ] 

Jeff Beagley commented on KAFKA-7157:
-------------------------------------

Never programmed java before, but I gave it my best shot; I basically just 
checked for null, and if so returned null from the transformation. I kept 
landing with the same error. This is using Avro with the schema below
{code:java}
Caused by: org.apache.kafka.connect.errors.DataException: Invalid value: null 
used for required field: "nullable_date", schema type: INT64
{code}
!https://www.freeformatter.com/3.4.0.2/img/minus.gif!
"type": "record","name": "Envelope","namespace": 
"mssql.dbo.test_dates_t","fields": 
[!https://www.freeformatter.com/3.4.0.2/img/minus.gif!{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name":
 "before","type": 
[!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"null",{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"type":
 "record","name": "Value","fields": 
[!https://www.freeformatter.com/3.4.0.2/img/minus.gif!{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name":
 "id","type": 
"int"},{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name": 
"non_nullable_date","type": 
{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"type": 
"long","connect.version": 1,"connect.name": 
"io.debezium.time.Timestamp"}},{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name":
 "nullable_date","type": 
[!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"null",{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"type":
 "long","connect.version": 1,"connect.name": 
"io.debezium.time.Timestamp"}],"default": null}],"connect.name": 
"mssql.dbo.test_dates_t.Value"}],"default": 
null},{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name": 
"after","type": 
[!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"null","Value"],"default":
 null},{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name": 
"source","type": {!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"type": 
"record","name": "Source","namespace": 
"io.debezium.connector.sqlserver","fields": 
[!https://www.freeformatter.com/3.4.0.2/img/minus.gif!{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name":
 "version","type": 
[!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"null","string"],"default":
 null},{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name": 
"connector","type": 
[!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"null","string"],"default":
 null},{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name": 
"name","type": 
"string"},{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name": 
"ts_ms","type": 
[!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"null","long"],"default": 
null},{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name": 
"change_lsn","type": 
[!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"null","string"],"default":
 null},{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name": 
"commit_lsn","type": 
[!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"null","string"],"default":
 null},{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name": 
"snapshot","type": 
[!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"null","boolean"],"default":
 null}],"connect.name": 
"io.debezium.connector.sqlserver.Source"}},{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name":
 "op","type": 
"string"},{!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"name": 
"ts_ms","type": 
[!https://www.freeformatter.com/3.4.0.2/img/minus.gif!"null","long"],"default": 
null}],"connect.name": "mssql.dbo.test_dates_t.Envelope"}

> Connect TimestampConverter SMT doesn't handle null values
> ---------------------------------------------------------
>
>                 Key: KAFKA-7157
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7157
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 0.10.2.0
>            Reporter: Randall Hauch
>            Assignee: Valeria Vasylieva
>            Priority: Major
>
> TimestampConverter SMT is not able to handle null values (in any versions), 
> so it's always trying to apply the transformation to the value. Instead, it 
> needs to check for null and use the default value for the new schema's field.
> {noformat}
> [2018-07-03 02:31:52,490] ERROR Task MySourceConnector-2 threw an uncaught 
> and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask) 
> java.lang.NullPointerException 
> at 
> org.apache.kafka.connect.transforms.TimestampConverter$2.toRaw(TimestampConverter.java:137)
>  
> at 
> org.apache.kafka.connect.transforms.TimestampConverter.convertTimestamp(TimestampConverter.java:440)
>  
> at 
> org.apache.kafka.connect.transforms.TimestampConverter.applyValueWithSchema(TimestampConverter.java:368)
>  
> at 
> org.apache.kafka.connect.transforms.TimestampConverter.applyWithSchema(TimestampConverter.java:358)
>  
> at 
> org.apache.kafka.connect.transforms.TimestampConverter.apply(TimestampConverter.java:275)
>  
> at 
> org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:38)
>  
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:435)
>  
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:264) 
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:182)
>  
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:150)
>  
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146) 
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190) 
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  
> at java.lang.Thread.run(Thread.java:748) 
> [2018-07-03 02:31:52,491] ERROR Task is being killed and will not recover 
> until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) 
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)