[
https://issues.apache.org/jira/browse/BEAM-9035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Van Boxel updated BEAM-9035:
---------------------------------
Description:
This is the first issue of a multipart commit: this ticket implements the basic
infrastructure of options on row and field.
Full explanation:
Introduce the concept of Options in Beam Schema’s to add extra context to
fields and schema. In contracts to metadata, options would be added to fields,
logical types and rows. In the options schema convertors can add
options/annotations/decorators that were in the original schema, this context
can be used in the rest of the pipeline for specific transformations or augment
the end schema in the target output.
Examples of options are:
* informational: like the source of the data, ...
* drive decisions further in the pipeline: flatten a row into another, rename
a field, ...
* influence something in the output: like cluster index, primary key, ...
* logical type information
And option is a key/typed value combination. The advantages of having the value
types is:
* Having strongly typed options would give a *portable way of Logical Types*
to have structured information that could be shared over different languages.
* This could keep the type intact when mapping from a formats that have
strongly typed options (example: Protobuf).
This is part of a multi ticket implementation. The following tickets are
related:
# Typed options for Row Schema and Fields
# Convert Proto Options to Beam Schema options
# Convert Avro extra information for Beam string options
# Replace meta data with Logical Type options
# Extract meta data in Calcite SQL to Beam options
# Extract meta data in Zeta SQL to Beam options
# Add java example of using option in a transform
This feature is discussed with Reuven Lax, Brian Hulette
was:
This is the first issue of a multipart commit:
Introduce the concept of Options in Beam Schema’s to add extra context to
fields and schema. In contracts to metadata, options would be added to fields,
logical types and rows. In the options schema convertors can add
options/annotations/decorators that were in the original schema, this context
can be used in the rest of the pipeline for specific transformations or augment
the end schema in the target output.
Examples of options are:
* informational: like the source of the data, ...
* drive decisions further in the pipeline: flatten a row into another, rename
a field, ...
* influence something in the output: like cluster index, primary key, ...
* logical type information
And option is a key/typed value combination. The advantages of having the value
types is:
* Having strongly typed options would give a *portable way of Logical Types*
to have structured information that could be shared over different languages.
* This could keep the type intact when mapping from a formats that have
strongly typed options (example: Protobuf).
This is part of a multi ticket implementation. The following tickets are
related:
# Typed options for Row Schema and Fields
# Convert Proto Options to Beam Schema options
# Convert Avro extra information for Beam string options
# Replace meta data with Logical Type options
# Extract meta data in Calcite SQL to Beam options
# Extract meta data in Zeta SQL to Beam options
# Add java example of using option in a transform
This feature is discussed with Reuven Lax, Brian Hulette
> Typed options for Row Schema and Fields
> ---------------------------------------
>
> Key: BEAM-9035
> URL: https://issues.apache.org/jira/browse/BEAM-9035
> Project: Beam
> Issue Type: Task
> Components: sdk-java-core
> Reporter: Alex Van Boxel
> Assignee: Alex Van Boxel
> Priority: Major
>
> This is the first issue of a multipart commit: this ticket implements the
> basic infrastructure of options on row and field.
> Full explanation:
> Introduce the concept of Options in Beam Schema’s to add extra context to
> fields and schema. In contracts to metadata, options would be added to
> fields, logical types and rows. In the options schema convertors can add
> options/annotations/decorators that were in the original schema, this context
> can be used in the rest of the pipeline for specific transformations or
> augment the end schema in the target output.
> Examples of options are:
> * informational: like the source of the data, ...
> * drive decisions further in the pipeline: flatten a row into another,
> rename a field, ...
> * influence something in the output: like cluster index, primary key, ...
> * logical type information
> And option is a key/typed value combination. The advantages of having the
> value types is:
> * Having strongly typed options would give a *portable way of Logical Types*
> to have structured information that could be shared over different languages.
> * This could keep the type intact when mapping from a formats that have
> strongly typed options (example: Protobuf).
> This is part of a multi ticket implementation. The following tickets are
> related:
> # Typed options for Row Schema and Fields
> # Convert Proto Options to Beam Schema options
> # Convert Avro extra information for Beam string options
> # Replace meta data with Logical Type options
> # Extract meta data in Calcite SQL to Beam options
> # Extract meta data in Zeta SQL to Beam options
> # Add java example of using option in a transform
> This feature is discussed with Reuven Lax, Brian Hulette
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
