[
https://issues.apache.org/jira/browse/BEAM-9035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Van Boxel updated BEAM-9035:
---------------------------------
Fix Version/s: (was: 2.20.0)
2.21.0
> BIP-1: Typed options for Row Schema and Fields
> ----------------------------------------------
>
> Key: BEAM-9035
> URL: https://issues.apache.org/jira/browse/BEAM-9035
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
> Reporter: Alex Van Boxel
> Assignee: Alex Van Boxel
> Priority: Major
> Fix For: 2.21.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> This is the first issue of a multipart commit: this ticket implements the
> basic infrastructure of options on row and field.
> Full explanation:
> Introduce the concept of Options in Beam Schema’s to add extra context to
> fields and schema. In contracts to metadata, options would be added to
> fields, logical types and rows. In the options schema convertors can add
> options/annotations/decorators that were in the original schema, this context
> can be used in the rest of the pipeline for specific transformations or
> augment the end schema in the target output.
> Examples of options are:
> * informational: like the source of the data, ...
> * drive decisions further in the pipeline: flatten a row into another,
> rename a field, ...
> * influence something in the output: like cluster index, primary key, ...
> * logical type information
> And option is a key/typed value combination. The advantages of having the
> value types is:
> * Having strongly typed options would give a *portable way of Logical Types*
> to have structured information that could be shared over different languages.
> * This could keep the type intact when mapping from a formats that have
> strongly typed options (example: Protobuf).
> This is part of a multi ticket implementation. The following tickets are
> related:
> # Typed options for Row Schema and Fields
> # Convert Proto Options to Beam Schema options
> # Convert Avro extra information for Beam string options
> # Replace meta data with Logical Type options
> # Extract meta data in Calcite SQL to Beam options
> # Extract meta data in Zeta SQL to Beam options
> # Add java example of using option in a transform
> This feature is discussed with Reuven Lax, Brian Hulette
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
