[
https://issues.apache.org/jira/browse/BEAM-13420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-13420:
-----------------------------------
Status: Open (was: Triage Needed)
> [Go SDK] Decoding a schema row doesn't respect field names from struct tags
> ---------------------------------------------------------------------------
>
> Key: BEAM-13420
> URL: https://issues.apache.org/jira/browse/BEAM-13420
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Reporter: Daniel Oliveira
> Priority: P2
>
> Will attempt to create a reproducible code snippet soon.
> For now, the basics of the bug are that when an element encoded as a Row via
> schemas gets decoded to a Go struct, it doesn't respect struct tags somewhere
> in the conversion process. More specifically, if I have an external transform
> that outputs a Row, and I take that as input to a native Go transform that
> accepts some Go struct "Foo", it will fail if the field names are different,
> even if the struct tags on Foo match the external row's field names.
> Example error message. The following error is caused by the WordCount and
> CorpusDate fields in the native struct not matching the field names
> "word_count" and "corpus_date" from the raw Row. The row gets decoded into a
> struct with the field names Word_count and Corpus_date, and ignoring the
> struct tags of the struct it's attempting to match:
> {noformat}
> panic: reflect: Call using struct { Word string "beam:\"word\""; Word_count
> int64 "beam:\"word_count\""; Corpus string "beam:\"corpus\""; Corpus_date
> int64 "beam:\"corpus_date\"" } as type struct { Word string "beam:\"word\"";
> WordCount int64 "beam:\"word_count\""; Corpus string "beam:\"corpus\"";
> CorpusDate int64 "beam:\"corpus_date\"" }
> Full error:
> while executing Process for Plan[process-bundle-descriptor-291]:
> 2: DataSink[S[ptransform-289@localhost:12371]]
> Coder:W;coder-315<KV;coder-316<int[varintz;c3];coder-317,string;coder-318>>!GWC
> 3: PCollection[pcollection-304] Out:[2]
> 4: ParDo[beam.addFixedKeyFn] Out:[2]
> 5: PCollection[pcollection-300] Out:[4]
> 6: ParDo[main.main.func1] Out:[5]
> 1: DataSource[S[ptransform-288@localhost:12371], 0]
> Coder:W;coder-310<LP;coder-311<R[struct { Word string "beam:\"word\"";
> Word_count int64 "beam:\"word_count\""; Corpus string "beam:\"corpus\"";
> Corpus_date int64 "beam:\"corpus_date\"" }]>>!GWC Out:6
> caused by:
> panic: reflect: Call using struct { Word string "beam:\"word\""; Word_count
> int64 "beam:\"word_count\""; Corpus string "beam:\"corpus\""; Corpus_date
> int64 "beam:\"corpus_date\"" } as type struct { Word string "beam:\"word\"";
> WordCount int64 "beam:\"word_count\""; Corpus string "beam:\"corpus\"";
> CorpusDate int64 "beam:\"corpus_date\"" } goroutine 39 [running]:
> runtime/debug.Stack()
> /usr/lib/google-golang/src/runtime/debug/stack.go:24 +0x65
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.callNoPanic.func1()
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/util.go:58
> +0xa5
> panic({0xe8f160, 0xc0003149e0})
> /usr/lib/google-golang/src/runtime/panic.go:1038 +0x215
> reflect.Value.call({0xc0003267e0, 0xc0003b3e10, 0x5be732}, {0x102e0e5, 0x4},
> {0xc0000fe3d8, 0x1, 0x679031})
> /usr/lib/google-golang/src/reflect/value.go:411 +0x1965
> reflect.Value.Call({0xc0003267e0, 0xc0003b3e10, 0x1}, {0xc0000fe3d8, 0x1,
> 0x1})
> /usr/lib/google-golang/src/reflect/value.go:339 +0xc5
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/util/reflectx.(*reflectFunc).Call(0xc000426060,
> {0xc0003326f0, 0x0, 0xf5ef00})
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/util/reflectx/call.go:87
> +0x59
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*invoker).initCall.func33({0x18e7a90,
> 0x1, 0x1}, 0xffdf3b645a1cac09)
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/fn_arity.go:229
> +0x7b
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*invoker).Invoke(0xc000477860,
> {0x1168a50, 0xc00033cdc0}, {0x18e7a90, 0x0, 0x1}, 0x203000, 0xc00015d660,
> {0x1934830, 0x0, ...})
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/fn.go:186
> +0x7a2
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*ParDo).invokeProcessFn(0xc0004420e0,
> {0x1168a50, 0xc00033cdc0}, {0x18e7a90, 0x1, 0x1}, 0x30, 0x1904be0)
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/pardo.go:316
> +0x146
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*ParDo).processSingleWindow(0xc0004420e0,
> 0xc00015d660)
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/pardo.go:166
> +0x4b
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*ParDo).processMainInput(0xc000348fc0,
> 0x1168a50)
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/pardo.go:146
> +0x9c
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*ParDo).ProcessElement(0xc0004420e0,
> {0x114d8a0, 0xc000426300}, 0xc0001f2540, {0x0, 0x0, 0x0})
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/pardo.go:132
> +0x1a5
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*DataSource).Process(0xc0000ec8c0,
> {0x1168a50, 0xc00033cd00})
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/datasource.go:174
> +0x4ab
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.callNoPanic({0x1168a50,
> 0xc00033cd00}, 0xc0002d6240)
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/util.go:62
> +0x6c
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*Plan).Execute(0xc000457030,
> {0x1168a50, 0xc00033cd00}, {0xc0003f07b0, 0x24}, {{0x115ee60, 0xc0002d61b0},
> {0x11631d8, 0xc0002d61e0}})
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/plan.go:118
> +0x3a5
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness.(*control).handleInstruction(0xc000452000,
> {0x1168938, 0xc00033e0f0}, 0xc0003fe550)
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:331
> +0xb6f
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness.Main.func3({0x1168938,
> 0xc00033e0f0}, 0xc0003fe550)
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:144
> +0x19d
> created by
> github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness.Main
>
> /usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:163
> +0xa8f{noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
