[
https://issues.apache.org/jira/browse/GOBBLIN-238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Tiwari updated GOBBLIN-238:
------------------------------------
Fix Version/s: 0.12
> Implement EnvelopePayloadExtractor and EnvelopePayloadDeserializer
> ------------------------------------------------------------------
>
> Key: GOBBLIN-238
> URL: https://issues.apache.org/jira/browse/GOBBLIN-238
> Project: Apache Gobblin
> Issue Type: Task
> Reporter: Zhixiong Chen
> Assignee: Zhixiong Chen
> Labels: Core:Converter
> Fix For: 0.12
>
>
> h3. Why
> The current implementation of EnvelopeSchemaConverter has several flaws:
> - Assumes top level payload schema field
> - Output record is the schema'ed payload but output schema is a String
> To address the issues and improve envelope schema conversion, the task
> implements two types of EnvelopeSchemaConverter: EnvelopePayloadExtractor and
> EnvelopePayloadDeserializer.
> h3. EnvelopePayloadExtractor
> This is a replacement of the deprecated `EvenlopeSchemaConverter`. Given an
> envelope record, the output schema will be the latest payload schema fetched
> from a kafka registry. The output record will be the deserialized payload
> with the latest schema
> h3. EnvelopePayloadDeserializer
> Given an envelope record, the output schema will set the payload field to
> have the latest schema fetched from a kafka registry and set the other fields
> as they are from the input schema. The output record will set the payload to
> be the deserialized object with the latest schema and set the other fields as
> they are from the input record
> h3. Configurations
> One configuration is required to set for any of the converters to work. It
> has no default value.
> {code:java}
> // The topic to fetch the latest schema of the payload from a kafka registry
> converter.envelopeSchemaConverter.payloadSchemaTopic=
> {code}
> The converter supports nested schema id
> {code:java}
> converter.envelopeSchemaConverter.schemaIdField="metadata.payloadSchemaId"
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
