[
https://issues.apache.org/jira/browse/BEAM-7996?focusedWorklogId=466360&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-466360
]
ASF GitHub Bot logged work on BEAM-7996:
----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Aug/20 17:47
Start Date: 04/Aug/20 17:47
Worklog Time Spent: 10m
Work Description: lostluck commented on a change in pull request #12426:
URL: https://github.com/apache/beam/pull/12426#discussion_r465223398
##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -855,10 +855,21 @@ message StandardCoders {
// BOOLEAN: beam:coder:bool:v1
// BYTES: beam:coder:bytes:v1
// ArrayType: beam:coder:iterable:v1 (always has a known length)
- // MapType: not yet a standard coder (BEAM-7996)
+ // MapType: not a standard coder, specification defined below.
// RowType: beam:coder:row:v1
// LogicalType: Uses the coder for its representation.
//
+ // The MapType is encoded by:
+ // - An INT32 representing the size of the map (N)
+ // - Followed by N interleaved keys and values, encoded with their
+ // corresponding coder.
+ //
+ // Nullable types in container types (ArrayType, MapType) are encoded by:
+ // - A one byte null indicator, 0x00 for null values, or 0x01 for present
+ // values.
+ // - For present values the null indicator is followed by the value
+ // encoded with it's corresponding coder.
+ //
Review comment:
@robertwb Your comment fed my own misunderstanding. It is possible to
declare a map in schemas as not having null/keys values, but not necessarily on
the SDK side.
Technically, there's no reason that the SDK can't use a non-nullable
containing version of the container if the Key and Value components are not
themselves marked as nillable. IIRC, the Java SDK *could* converted
ImmutableMaps or similar into just non-nullable Key and non-nullable Value
types.
The issue as I'm understanding it is that the limitation is on the SDK
Language side, rather than the schema specification side, as discussed the
schemas fields can individually have their nullable bits set.
Eg. Go doesn't have this ambiguity for map types.
On the other hand, in Go, Iterable/array types which will be represented by
slices *will* have this ambiguity when used as a field, as they can be nil, and
could also still be pointers to said reference types. That ambiguity is well
known enough that pointers to reference types (maps, slices, chans..) are
strongly discouraged in idiomatic Go.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 466360)
Time Spent: 6h 40m (was: 6.5h)
> Add support for remaining data types in python RowCoder
> --------------------------------------------------------
>
> Key: BEAM-7996
> URL: https://issues.apache.org/jira/browse/BEAM-7996
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
> Reporter: Brian Hulette
> Assignee: Brian Hulette
> Priority: P2
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> In the initial [python RowCoder
> implementation|https://github.com/apache/beam/pull/9188] we only added
> support for the data types that already had coders in the Python SDK. We
> should add support for the remaining data types that are not currently
> supported:
> * INT8 (ByteCoder in Java)
> * INT16 (BigEndianShortCoder in Java)
> * FLOAT (FloatCoder in Java) (Note: doubles are supported, this is
> specifically for single-precision)
> * --BOOLEAN (standard beam:coder:bool:v1, BooleanCoder in Java)--
> * --BYTES (standard beam:coder:bytes:v1, ByteArrayCoder in Java)--
> * Map (MapCoder in Java)
> We might consider making those coders standard so they can be tested
> independently from RowCoder in standard_coders.yaml. Or, if we don't do that
> we should probably add a more robust testing framework for RowCoder itself,
> because it will be challenging to test all of these types as part of the
> RowCoder tests in standard_coders.yaml.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
