Feroze Daud created AVRO-3026:
---------------------------------
Summary: Allow custom annotations in IDL files and support
translating them to AVSC Avro.
Key: AVRO-3026
URL: https://issues.apache.org/jira/browse/AVRO-3026
Project: Apache Avro
Issue Type: New Feature
Components: spec
Affects Versions: 1.10.1, 1.9.2, 1.9.1, 1.9.0
Reporter: Feroze Daud
h2. Introduction
Our company has standardized on Avro schemas for all Data intestion and
storage. As part of this, and to satisfy CCPA, we need to be able to tag the
records and fields appropriately if they have PI, or Non PI information, etc.
Avro AVSC files, being valid json, can easily be modified to add tags that will
be used by downstream processors, and also wont interfere with Avro itself ( to
generate POJO, serialization, deserialization etc)
One such key we chose is simply called *tags*. It's example usage is shown
below.
{code:java}
{
"type": "record",
"name": "PropertyOwner",
"namespace": "com.acme.Property",
"tags": ["PI", "PII" ],
"fields": [
{
"name": "FullName",
"type": "string",
"tags": ["Name"]
},
{
"name": "PhoneNumber",
"type": "string",
"tags": ["Phone"]
}],
}{code}
These tags can be processed by downstream processors and the data landing in
datalake, or database can be tagged appropriately.
h2. Problem Description
While tagging will work fine for AVSC because adding extra fields doesnt make
it invalid, we will have a problem when using IDL to author schemas. IDL spec
does not allow a way to add extra tags that are copied over to the Avro schema.
h2. Proposal
I propose that we allow a special *@annotation* tag . And, this tag can be
applied to records and fields. Whatever is in this annotation should be copied
verbatim to the output AVSC.
For eg:
{code:java}
@annotation("tags", "[\"PI\", \"Non PI\"]"
record Employee {
@annotation("tags", "[\"Name\"]"
string fullName;
boolean active = true;
long salary;
@annotation("tags", "[\"Phone\"]"
string phone;
} {code}
would generate an avro schema as folllows:
{code:java}
{
"type": "record",
"name": "Employee",
"tags": ["PI", "PII" ],
"fields": [
{
"name": "FullName",
"type": "string",
"tags": ["Name"]
},
{
"name": "PhoneNumber",
"type": "string",
"tags": ["Phone"]
}],
}{code}
As you can see, we dont need to support any wellformed JSONness in the
*@annotation* . It just takes a string and we just render it into the output
json.
@annotation("foo", "[\"bar\"]") -> "tags": ["bar"]
@annotation("foo", "\{\"bar\": \"jar\"}") -> "tags": {"bar": "jar"}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
