[
https://issues.apache.org/jira/browse/AVRO-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339950#comment-17339950
]
Feroze Daud commented on AVRO-3026:
-----------------------------------
Hi!
Finally had a chance to try it out. So the `@tags` annotation works. But there
doesnt seem to be a way to add a `doc` tag as allowed by AVSC schema to the
output file.
How do we achieve that?
For eg, this causes an error...
{noformat}
record employee {
string @doc("employee name") name;
boolean active;
long salary;
}
{noformat}
Output:
{noformat}
$ java -jar ~/DevTools/avro-tools-1.10.1.jar idl employee.avdl | jq .$ java
-jar ~/DevTools/avro-tools-1.10.1.jar idl employee.avdl | jq .Exception in
thread "main" org.apache.avro.AvroRuntimeException: Can't set reserved
property: doc at
org.apache.avro.JsonProperties.addProp(JsonProperties.java:281) at
org.apache.avro.JsonProperties.access$000(JsonProperties.java:121) at
org.apache.avro.JsonProperties$1.addProp(JsonProperties.java:127) at
org.apache.avro.util.internal.Accessor.addProp(Accessor.java:101) at
org.apache.avro.compiler.idl.Idl.VariableDeclarator(Idl.java:664) at
org.apache.avro.compiler.idl.Idl.FieldDeclaration(Idl.java:606) at
org.apache.avro.compiler.idl.Idl.RecordDeclaration(Idl.java:569) at
org.apache.avro.compiler.idl.Idl.NamedSchemaDeclaration(Idl.java:153) at
org.apache.avro.compiler.idl.Idl.ProtocolBody(Idl.java:402) at
org.apache.avro.compiler.idl.Idl.ProtocolDeclaration(Idl.java:227) at
org.apache.avro.compiler.idl.Idl.CompilationUnit(Idl.java:117) at
org.apache.avro.tool.IdlTool.run(IdlTool.java:61) at
org.apache.avro.tool.Main.run(Main.java:67) at
org.apache.avro.tool.Main.main(Main.java:56) {noformat}
> Allow custom annotations in IDL files and support translating them to AVSC
> Avro.
> --------------------------------------------------------------------------------
>
> Key: AVRO-3026
> URL: https://issues.apache.org/jira/browse/AVRO-3026
> Project: Apache Avro
> Issue Type: New Feature
> Components: spec
> Affects Versions: 1.9.0, 1.9.1, 1.9.2, 1.10.1
> Reporter: Feroze Daud
> Priority: Major
>
> h2. Introduction
> Our company has standardized on Avro schemas for all Data intestion and
> storage. As part of this, and to satisfy CCPA, we need to be able to tag the
> records and fields appropriately if they have PI, or Non PI information, etc.
> Avro AVSC files, being valid json, can easily be modified to add tags that
> will be used by downstream processors, and also wont interfere with Avro
> itself ( to generate POJO, serialization, deserialization etc)
> One such key we chose is simply called *tags*. It's example usage is shown
> below.
> {code:java}
> {
> "type": "record",
> "name": "PropertyOwner",
> "namespace": "com.acme.Property",
> "tags": ["PI", "PII" ],
> "fields": [
> {
> "name": "FullName",
> "type": "string",
> "tags": ["Name"]
> },
> {
> "name": "PhoneNumber",
> "type": "string",
> "tags": ["Phone"]
> }],
> }{code}
>
> These tags can be processed by downstream processors and the data landing in
> datalake, or database can be tagged appropriately.
>
> h2. Problem Description
> While tagging will work fine for AVSC because adding extra fields doesnt make
> it invalid, we will have a problem when using IDL to author schemas. IDL spec
> does not allow a way to add extra tags that are copied over to the Avro
> schema.
>
> h2. Proposal
> I propose that we allow a special *@annotation* tag . And, this tag can be
> applied to records and fields. Whatever is in this annotation should be
> copied verbatim to the output AVSC.
> For eg:
> {code:java}
> @annotation("tags", "[\"PI\", \"Non PI\"]"
> record Employee {
> @annotation("tags", "[\"Name\"]"
> string fullName;
> boolean active = true;
> long salary;
> @annotation("tags", "[\"Phone\"]"
> string phone;
> } {code}
>
> would generate an avro schema as folllows:
>
> {code:java}
> {
> "type": "record",
> "name": "Employee",
> "tags": ["PI", "PII" ],
> "fields": [
> {
> "name": "FullName",
> "type": "string",
> "tags": ["Name"]
> },
> {
> "name": "PhoneNumber",
> "type": "string",
> "tags": ["Phone"]
> }],
> }{code}
>
> As you can see, we dont need to support any wellformed JSONness in the
> *@annotation* . It just takes a string and we just render it into the output
> json.
> @annotation("foo", "[\"bar\"]") -> "tags": ["bar"]
> @annotation("foo", "\{\"bar\": \"jar\"}") -> "tags": {"bar": "jar"}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
