Robin Moffatt created KAFKA-10865:
-------------------------------------
Summary: Improve trace-logging for Transformations (including
Predicates)
Key: KAFKA-10865
URL: https://issues.apache.org/jira/browse/KAFKA-10865
Project: Kafka
Issue Type: Improvement
Components: KafkaConnect
Reporter: Robin Moffatt
I've been spending [a bunch of time poking around
SMTs|https://rmoff.net/categories/twelvedaysofsmt/] recently, and one common
challenge I've had is being able to debug when things don't behave as I expect.
I know that there is the {{TransformationChain}} logger, but this only gives
(IIUC) the input record
{code:java}
[2020-12-17 09:38:58,057] TRACE [sink-simulator-day12-00|task-0] Applying
transformation io.confluent.connect.transforms.Filter$Value to
SinkRecord{kafkaOffset=10551, timestampType=CreateTime}
ConnectRecord{topic='day12-sys01', kafkaPartition=0,
key=2c2ceb9b-8b31-4ade-a757-886ebfb7a398, keySchema=Schema{STRING},
value=Struct{units=16,product=Founders Breakfast
Stout,amount=30.41,txn_date=Sat Dec 12 18:21:18 GMT 2020,source=SYS01},
valueSchema=Schema{io.mdrogalis.Gen0:STRUCT}, timestamp=1608197938054,
headers=ConnectHeaders(headers=)}
(org.apache.kafka.connect.runtime.TransformationChain:47)
{code}
I think it would be really useful to also have trace level logging that
included:
- the _output_ of *each* transform
- the evaluation and result of any `predicate`s
I have been using
{{com.github.jcustenborder.kafka.connect.simulator.SimulatorSinkConnector}}
which is really useful for seeing the final record:
{code:java}
[2020-12-17 09:38:58,057] INFO [sink-simulator-day12-00|task-0]
record.value=Struct{units=16,product=Founders Breakfast
Stout,amount=30.41,txn_date=Sat Dec 12 18:21:18 GMT 2020,source=SYS01}
(com.github.jcustenborder.kafka.connect.simulator.SimulatorSinkTask:50)
{code}
But doesn't include things like topic name (which is often changed by common
SMTs)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
