[
https://issues.apache.org/jira/browse/BEAM-7009?focusedWorklogId=484209&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484209
]
ASF GitHub Bot logged work on BEAM-7009:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Sep/20 21:19
Start Date: 14/Sep/20 21:19
Worklog Time Spent: 10m
Work Description: lostluck commented on a change in pull request #12588:
URL: https://github.com/apache/beam/pull/12588#discussion_r488226218
##########
File path: sdks/go/pkg/beam/core/runtime/exec/coder.go
##########
@@ -434,6 +600,267 @@ func convertIfNeeded(v interface{}, allocated *FullValue)
*FullValue {
return allocated
}
+type iterableEncoder struct {
+ t reflect.Type
+ enc ElementEncoder
+}
+
+func (c *iterableEncoder) Encode(val *FullValue, w io.Writer) error {
+ // Do a reflect, get the length.
+ rv := reflect.ValueOf(val.Elm)
+ size := rv.Len()
+ if err := coder.EncodeInt32((int32)(size), w); err != nil {
+ return err
+ }
+ var e FullValue
+ for i := 0; i < size; i++ {
+ e.Elm = rv.Index(i).Interface()
+ err := c.enc.Encode(&e, w)
+ if err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+type iterableDecoder struct {
+ t reflect.Type
+ dec ElementDecoder
+}
+
+func (c *iterableDecoder) DecodeTo(r io.Reader, fv *FullValue) error {
+ // (1) Read count prefixed encoded data
+
+ size, err := coder.DecodeInt32(r)
+ if err != nil {
+ return err
+ }
+ n := int(size)
+ switch {
+ case n >= 0:
+ rv, err := c.decodeToSlice(int(n), r)
+ if err != nil {
+ return err
+ }
+ *fv = FullValue{Elm: rv.Interface()}
+ return nil
+ case n == -1:
+ rv := reflect.MakeSlice(c.t, 0, 0)
+ chunk, err := coder.DecodeVarInt(r)
+ if err != nil {
+ return err
+ }
+ for chunk != 0 {
+ rvi, err := c.decodeToSlice(int(chunk), r)
+ if err != nil {
+ return err
+ }
+ rv = reflect.AppendSlice(rv, rvi)
+ chunk, err = coder.DecodeVarInt(r)
+ if err != nil {
+ return err
+ }
+ }
+ *fv = FullValue{Elm: rv.Interface()}
+ }
+
+ return nil
+}
+
+func (c *iterableDecoder) decodeToSlice(n int, r io.Reader) (reflect.Value,
error) {
+ var e FullValue
+ rv := reflect.MakeSlice(c.t, n, n)
+ for i := 0; i < int(n); i++ {
+ err := c.dec.DecodeTo(r, &e)
+ if err != nil {
+ return reflect.Value{}, err
+ }
+ if e.Elm != nil {
+ rv.Index(i).Set(reflect.ValueOf(e.Elm))
+ } else {
+ rv.Index(i).Set(reflect.ValueOf(e.Windows[0]))
Review comment:
In this case it's to set it to the 0th window element. In this case it's
a small hack to support "window's as element values" which is only necessary to
dramatically simplify the yaml tests which actually test the coder values here.
Deserves a comment to explain it, as it is a hack.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 484209)
Time Spent: 4h 10m (was: 4h)
> Add Go SDK test for Standard Coders using the yaml data.
> --------------------------------------------------------
>
> Key: BEAM-7009
> URL: https://issues.apache.org/jira/browse/BEAM-7009
> Project: Beam
> Issue Type: Bug
> Components: beam-model, sdk-go
> Reporter: Robert Burke
> Assignee: Robert Burke
> Priority: P1
> Labels: stale-P2
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> The Go SDK doesn't currently do this validation, against the standard yaml
> file. [1]
> The Java and Python equivalents of the test can be found from here [2].
>
> Care would need to be taken so that Beam Go SDK users (such as they are)
> aren't forced to run them, and not have the yaml file to read. I'd suggest
> putting it with the integration tests [3].
> The other thing of note is that the Go SDK has no notion of "nested" vs
> "unnested" coders. All coders are "nested" in the Go SDK, and should have
> their lengths prefixed to them as appropriate.
> 1:
> [https://github.com/apache/beam/blob/master/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml]
> 2:
> [https://github.com/apache/beam/search?q=standard_coders.yaml&unscoped_q=standard_coders.yaml]
> 3: [https://github.com/apache/beam/tree/master/sdks/go/test
> |https://github.com/apache/beam/tree/master/sdks/go/test]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
