Tomas Roos created BEAM-5350:
--------------------------------
Summary: Running autocomplete.go on dataflow fails
Key: BEAM-5350
URL: https://issues.apache.org/jira/browse/BEAM-5350
Project: Beam
Issue Type: Bug
Components: sdk-go
Affects Versions: Not applicable
Reporter: Tomas Roos
Assignee: Henning Rohde
I'm in the process as a external developer make sure that all examples are
runnable on both direct and the dataflow runner as its crucial for people
onboarding this project.
I've visted the projects before and some are runnable, some probably where
previously, and some are def not runnable.
So I started top down today, in order to make autocomplete.go run on dataflow
as well as the direct runner i changed the input in order to make it platform
independent instead of pointing to a local file.The reading of the source from
the public cloud storage went fine but it fails to run the top.Largest
anonymous less function (ran on id: go-job-1-1536575613531078735) failed with
RESP: instruction_id: "-205" error: "Invalid bundle desc: decode: bad userfn:
bad struct encoding: failed to decode data: decode: failed to find symbol
main.main.func1: main.main.func1 not found. Use runtime.RegisterFunction in
unit tests" register: < >
So in order to fix this I introduced the local func called lessFn and
registered in the init process. This though now instead when running
go run autocomplete.go --project fair-app-213019 --runner dataflow
--staging_location=gs://fair-app-213019/staging-test2
--worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
fails with
2018/09/10 13:37:10 Running autocomplete
Unable to encode combiner for lifting: failed to encode custom coder: bad
underlying type: bad field type: bad element: unencodable type: interface
{}2018/09/10 13:37:10 Using running binary as worker binary:
'/tmp/go-build157286122/b001/exe/autocomplete'
2018/09/10 13:37:10 Staging worker binary:
/tmp/go-build157286122/b001/exe/autocomplete
And I know this is when invoking the top.Largest since I've removed the piece
of code and then the job runs fine, could you please point me in the right
direction why my local func is not encoable as a interface {} and I will of
course happily send a PR when this is working on direct and the dataflow direct
so I can move on to the other examples
(All changes can be seen here)
[https://github.com/apache/beam/compare/master...ptomasroos:make-autocomplete-dataflowable?expand=1]
https://github.com/apache/beam/blob/master/sdks/go/examples/complete/autocomplete/autocomplete.go#L63
2018/09/10 13:37:10 Running autocomplete
Unable to encode combiner for lifting: failed to encode custom coder: bad
underlying type: bad field type: bad element: unencodable type: interface
{}2018/09/10 13:37:10 Using running binary as worker binary:
'/tmp/go-build157286122/b001/exe/autocomplete'
2018/09/10 13:37:10 Staging worker binary:
/tmp/go-build157286122/b001/exe/autocomplete
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
