Robert Burke created BEAM-11095:
-----------------------------------
Summary: Improve error messages from Invalid element types in
emitter signatures
Key: BEAM-11095
URL: https://issues.apache.org/jira/browse/BEAM-11095
Project: Beam
Issue Type: Improvement
Components: sdk-go
Reporter: Robert Burke
If a DoFn's `ProcessElement`uses an emitter `func(MyType)` where `MyType` is
not a concrete type, (such as having function type fields, which are not valid)
the pipeline can fail with incomprehensible errors that don't explain the
problem correctly. eg.
{{panic: inserting ParDo in scope *mypackage.studyGenerationElement
constructing DoFn
invalid DoFn
caused by:
method ProcessElement invalid
caused by:
bad parameter type for reflect.methodValueCall:
func(*mypackage.studyGenerationElement)
goroutine 740 [running]:
panic(0x55e4c34f3e80, 0xc005952680)
go/gc/src/runtime/panic.go:1066+0x47e fp=0xc0061a18c8 sp=0xc0061a17d8
pc=0x55e49ff4b67e
testing.tRunner.func1.1(0x55e4c34f3e80, 0xc005952680)
go/gc/src/testing/testing.go:1076 +0x30d fp=0xc0061a1980 sp=0xc0061a18c8
pc=0x55e4a097254d
testing.tRunner.func1(0xc005046900)
go/gc/src/testing/testing.go:1079 +0x41a fp=0xc0061a1a28 sp=0xc0061a1980
pc=0x55e4a097297a
runtime.call32(0x0, 0x55e4c3f0e388, 0xc0050dbf38, 0x800000008)
go/gc/src/runtime/asm_amd64.s:540 +0x40 fp=0xc0061a1a58 sp=0xc0061a1a28
pc=0x55e49ff86fc0
runtime.reflectcallSave(0xc0061a1b88, 0x55e4c3f0e388, 0xc0050dbf38,
0x55e400000008)
go/gc/src/runtime/panic.go:881 +0x5a fp=0xc0061a1a88 sp=0xc0061a1a58
pc=0x55e49ff4b13a
runtime.runOpenDeferFrame(0xc0035bc180, 0xc0050dbef0, 0xc0061a1c08)
go/gc/src/runtime/panic.go:855 +0x2cd fp=0xc0061a1b18 sp=0xc0061a1a88
pc=0x55e49ff4afed
panic(0x55e4c34f3e80, 0xc005952680)
go/gc/src/runtime/panic.go:969 +0x175 fp=0xc0061a1c08 sp=0xc0061a1b18
pc=0x55e49ff4b375
...pkg/beam/beam.MustN(...)...pkg/beam/util.go:104
...pkg/beam/beam.ParDo(0xc00507d9e0, 0xc0052a7b30, 0x55e4c30faae0,
0xc0053244e0, 0xc0053245a0, 0x0, 0x0, 0x0, 0x0)
...pkg/beam/pardo.go:358 +0x135 fp=0xc0061a1c80 sp=0xc0061a1c08
pc=0x55e4a0f3cc95
}}
In particular this message is generated by
[funcx.New|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/funcx/fn.go#L323]
and occurs because we don't maintain an error state when trying to analyze the
DoFn.
We would catch most errors like this if we provide a justification for it
better.
This will likely mean reimplementing the IsConcrete function and similar in
terms of errors with the explanations, and having a justification handler to
return that error from funcx.New if the type is invalid. This is subtle as
while most errors are due to a type not being concrete, but the type might be
nested in a emitter or iterator function.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
