azach opened a new issue, #28959:
URL: https://github.com/apache/beam/issues/28959
### What happened?
Running into the following panic when attempting a partitioned read on
Dataflow:
```golang
spannerRows := spannerio.Query(s, *spannerConnStr, "select id from table",
reflect.TypeOf(SpannerEntity{}), spannerio.WithBatching(true))
```
Error seen on Dataflow:
```
Full error:
while executing Process for Plan[s01-10]:
1: DataSource[S[ptransform-7@localhost:12371], 0] Out:8
Coder:W;coder-35<bytes;coder-36>!GWC
8: ParDo[spannerio.generatePartitionsFn] Out:[7] Sig: func(context.Context,
[]uint8, func(spannerio.partitionedRead)) error, SideInputs: []
7: PCollection[s01-10-pcollection-15] Out:[6]
6: SDF.PairWithRestriction[spannerio.readBatchFn] UID:6 Out:[5]
5: PCollection[s01-10-pcollection-18] Out:[4]
4: SDF.SplitAndSizeRestrictions[spannerio.readBatchFn] UID:4 Out:[2]
3: PCollection[s01-10-pcollection-25] Out:[2]
2: DataSink[S[ptransform-8@localhost:12371]]
Coder:W;coder-41<KV;coder-42<KV;coder-43<LP;coder-44<R[spannerio.partitionedRead]>,KV;coder-45<offsetrange.Restriction[offsetrange.Restriction;c4];coder-46,bool;coder-47>>,double;coder-48>>!GWC
caused by:
panic: runtime error: invalid memory address or nil pointer dereference
goroutine 41 [running]:
runtime/debug.Stack()
runtime/debug/stack.go:24 +0x5e
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.callNoPanic.func1()
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/util.go:58
+0x94
panic({0x1b94d60?, 0x4011170?})
runtime/panic.go:914 +0x21f
cloud.google.com/go/spanner.(*Client).BatchReadOnlyTransactionFromID(0x0,
{{0xc0009f3300, 0x3e, 0x40}, {0xc000ad3cc0, 0xa0}, {0x30c20288, 0xedcb9b466,
0x4046f80}})
cloud.google.com/go/[email protected]/client.go:464 +0x57
github.com/apache/beam/sdks/v2/go/pkg/beam/io/spannerio.(*readBatchFn).CreateInitialRestriction(0xc000644fb0?,
{{{0xc0009f3300, 0x3e, 0x40}, {0xc000ad3cc0, 0xa0}, {0x30c20288, 0xedcb9b466,
0x4046f80}}, 0xc000b22d50})
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/spannerio/read_batch.go:70
+0x6d
reflect.Value.call({0x1ace740?, 0xc00043bf80?, 0x452909?}, {0x22605ba, 0x4},
{0xc000b74ba0, 0x1, 0x530ccb?})
reflect/value.go:596 +0xce7
reflect.Value.Call({0x1ace740?, 0xc00043bf80?, 0x99?}, {0xc000b74ba0?,
0xc0006c60e8?, 0xc000670c80?})
reflect/value.go:380 +0xb9
github.com/apache/beam/sdks/v2/go/pkg/beam/core/util/reflectx.(*reflectFunc).Call(0xc0001686c0,
{0xc000672440?, 0xc000b06240?, 0x246bf60?})
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/util/reflectx/call.go:87
+0x53
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*cirInvoker).initCallFn.func7()
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/sdf_invokers_arity.go:78
+0x32
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*cirInvoker).Invoke(0x0?,
{0x24a30d0?, 0xc000671d00?}, 0x1de8f00?)
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/sdf_invokers.go:84
+0xff
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*PairWithRestriction).ProcessElement(0xc000665050,
{0x24a30d0, 0xc000671d00}, 0xc0002a44d0, {0x0, 0x0, 0x0})
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/sdf.go:93
+0x5e
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*PCollection).ProcessElement(0xc00065f290?,
{0x24a30d0?, 0xc000671d00?}, 0x1?, {0x0?, 0x0?, 0x0?})
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/pcollection.go:100
+0x151
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*emitValue).invoke(0xc00052ce00,
{0xc000b74b88, 0x1, 0x101?})
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/emit.go:137
+0x2fb
github.com/apache/beam/sdks/v2/go/pkg/beam/io/spannerio.(*generatePartitionsFn).ProcessElement(0xc00052c380,
{0x24a30d0, 0xc000671d00}, {0xc0006ccd20?, 0xc000646920?, 0x40e45a?},
0xc00068f980)
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/spannerio/generate_partitions.go:96
+0x2f3
github.com/apache/beam/sdks/v2/go/pkg/beam/register.registerDoFn3x1StructWrappersAndFuncs[...].func2.1({0x40764e0,
0x0, 0x0}, 0xf20000c00067fd01)
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/register/register.go:3382
+0x78
github.com/apache/beam/sdks/v2/go/pkg/beam/register.(*caller3x1[...]).Call3x1(0x2529200,
{0x1e0a160, 0xc000671d00?}, {0x1a5b3c0, 0xc00069e918?}, {0x1a9b660,
0xc00068f980?})
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/register/register.go:3371
+0xa5
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*invoker).initCall.func13({0x3,
0x1, 0x1, 0x0, 0x0}, {0x4010b00, 0x1, 0x1}, 0x445ce0?)
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/fn_arity.go:109
+0xa9
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*invoker).invokeWithOpts(0xc0000fadc0,
{0x24a30d0?, 0xc000671d00}, {0x3, 0x1, 0x1, 0x0, 0x0}, {0x4010b00, 0x1, ...},
...)
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/fn.go:297
+0xf90
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*ParDo).invokeProcessFn(0xc000674000,
{0x24a30d0, 0xc000671d00}, {0x3, 0x1, 0x1, 0x0, 0x0}, {0x4010b00, 0x1, ...},
...)
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/pardo.go:508
+0x29f
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*ParDo).processSingleWindow(0xc000674000,
0xc0007001b0)
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/pardo.go:194
+0x8f
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*ParDo).processMainInput(0x0?,
0x24a30d0?)
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/pardo.go:172
+0x7f
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*ParDo).ProcessElement(0xc000674000,
{0x246bdc0?, 0xc00069e750?}, 0xc00052db90, {0x0, 0x0, 0x0})
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/pardo.go:156
+0x129
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*DataSource).Process.func1(0xc00069e750,
{0xc000647328?, 0xc0006c62c0?})
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/datasource.go:240
+0x223
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*DataSource).process(0xc0006302c0,
{0x24a30d0, 0xc00043a680}, 0xc000647618, 0xc0006475f8)
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/datasource.go:146
+0x4d3
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*DataSource).Process(0xc0006302c0,
{0x24a30d0, 0xc00043a680})
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/datasource.go:214
+0x20a
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*Plan).Execute.func2({0x24a30d0?,
0xc00043a680?})
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/plan.go:151
+0x3b
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.callNoPanic({0x24a30d0?,
0xc00043a680?}, 0x418488?)
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/util.go:62
+0x62
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*Plan).Execute(0xc00065f5f0,
{0x24a30d0, 0xc00043a680}, {0xc000350300, 0x24}, {{0x24a11a0?, 0xc000670500?},
{0x24f2bd8?, 0xc0006654d0?}})
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/exec/plan.go:150
+0x3b8
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness.(*control).handleInstruction(0xc000436000,
{0x24a2ad8?, 0xc000310f30?}, 0xc0004fc460)
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/harness/harness.go:409
+0x9f8
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness.MainWithOptions.func4({0x24a2ad8?,
0xc000310f30?}, 0xc000106900?)
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/harness/harness.go:195
+0x74
created by
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness.MainWithOptions
in goroutine 1
github.com/apache/beam/sdks/[email protected]/go/pkg/beam/core/runtime/harness/harness.go:213
+0xede
```
This collection works when batching is disabled. I've also confirmed this is
a root partitionable query (it's also just a simple read from the table).
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [X] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [X] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
