Hi Aaron,
You need to define the schema for the "data" field as a nested record.
{
"name": "data",
"type": {
"type": "record",
"name": "nestedData",
"fields": [
{"name": "message_type", "type": "string"},
{"name": "message_time", "type": "string"},
{"name": "file_ingest_time", "type": "string"}
]
}
}
Best Regards,
Lehel
________________________________
From: Aaron Rich <[email protected]>
Sent: Wednesday, November 22, 2023 7:30
To: [email protected] <[email protected]>; [email protected]
<[email protected]>
Subject: RecordReader/Writer with nested json resulting in MapRecord[] statement
Hello,
I'm trying to use the record writer/writers to take a json defined event and
write it to postgres database. The event follows the CloudEvent definition and
has a data element that is JSON structured:
{
"data" : {
"message_type" : "V5",
"message_time" : "1614597071000",
"file_ingest_time": "1682018983320"
},
"spec_version" : "1.0.2",
"subject" : "1969",
"data_schema" : "URI://something",
"id" : "80e12364-831e-446a-a260-86bccc469f25",
"source" : "service",
"time" : "2023-11-22T00:03:03Z",
"type" : "V5",
"data_content_type" : "application/json"
}
I'm trying to write into a postgres db table that has each field as a column
and data as a JSON type column.
When I try to write the the database with PutDatabaseRecord, I'm getting
MapRecord sting that is trying to be written into the data column:
MapRecord[{message_type=V5,
message_time=1614597071000,file_ingest_time=1682018983320}]
I _think_ the issue is with the schema I'm defining:
{
"type": "record",
"name": "eventSchema0",
"fields" : [
{"name":"data", "type": "string"},
{"name":"spec_version", "type": "string"},
{"name":"subject", "type": "string"},
{"name":"data_schema", "type": "string"},
{"name":"id", "type": "string"},
{"name":"source", "type": "string"},
{"name":"time", "type": "string"},
{"name":"type", "type": "string"},
{"name":"data_content_type", "type": "string"}
]
}
What do I need to change so the JSON structure in the "data" field can be
properly written using a record based processor?
Thanks for the help.
-Aaron
