[
https://issues.apache.org/jira/browse/BEAM-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomas Roos updated BEAM-5350:
-----------------------------
Description:
I'm in the process as a external developer make sure that all examples are
runnable on both direct and the dataflow runner as its crucial for people
onboarding this project.
I've visted the projects before and some are runnable, some probably where
previously, and some are def not runnable.
So I started top down today, in order to make autocomplete.go run on dataflow
as well as the direct runner i changed the input in order to make it platform
independent instead of pointing to a local file.The reading of the source from
the public cloud storage went fine but it fails to run the top.Largest
anonymous less function (ran on id: go-job-1-1536575613531078735) failed with
{code:java}
RESP: instruction_id: "-205" error: "Invalid bundle desc: decode: bad userfn:
bad struct encoding: failed to decode data: decode: failed to find symbol
main.main.func1: main.main.func1 not found. Use runtime.RegisterFunction in
unit tests" register: < >
{code}
[https://github.com/apache/beam/blob/master/sdks/go/examples/complete/autocomplete/autocomplete.go#L63]
So in order to fix this I introduced the local func called lessFn and
registered in the init process. This though now instead when running
{code:java}
go run autocomplete.go --project fair-app-213019 --runner dataflow
--staging_location=gs://fair-app-213019/staging-test2
--worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
{code}
fails with
{code:java}
2018/09/10 13:37:10 Running autocomplete
Unable to encode combiner for lifting: failed to encode custom coder: bad
underlying type: bad field type: bad element: unencodable type: interface
{}2018/09/10 13:37:10 Using running binary as worker binary:
'/tmp/go-build157286122/b001/exe/autocomplete'
2018/09/10 13:37:10 Staging worker binary:
/tmp/go-build157286122/b001/exe/autocomplete{code}
And I know this is when invoking the top.Largest since I've removed the piece
of code and then the job runs fine, could you please point me in the right
direction why my local func is not encoable as a interface {} and I will of
course happily send a PR when this is working on direct and the dataflow direct
so I can move on to the other examples
(All changes can be seen here)
[https://github.com/apache/beam/compare/master...ptomasroos:make-autocomplete-dataflowable?expand=1]
was:
I'm in the process as a external developer make sure that all examples are
runnable on both direct and the dataflow runner as its crucial for people
onboarding this project.
I've visted the projects before and some are runnable, some probably where
previously, and some are def not runnable.
So I started top down today, in order to make autocomplete.go run on dataflow
as well as the direct runner i changed the input in order to make it platform
independent instead of pointing to a local file.The reading of the source from
the public cloud storage went fine but it fails to run the top.Largest
anonymous less function (ran on id: go-job-1-1536575613531078735) failed with
RESP: instruction_id: "-205" error: "Invalid bundle desc: decode: bad userfn:
bad struct encoding: failed to decode data: decode: failed to find symbol
main.main.func1: main.main.func1 not found. Use runtime.RegisterFunction in
unit tests" register: < >
So in order to fix this I introduced the local func called lessFn and
registered in the init process. This though now instead when running
go run autocomplete.go --project fair-app-213019 --runner dataflow
--staging_location=gs://fair-app-213019/staging-test2
--worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
fails with
2018/09/10 13:37:10 Running autocomplete
Unable to encode combiner for lifting: failed to encode custom coder: bad
underlying type: bad field type: bad element: unencodable type: interface
{}2018/09/10 13:37:10 Using running binary as worker binary:
'/tmp/go-build157286122/b001/exe/autocomplete'
2018/09/10 13:37:10 Staging worker binary:
/tmp/go-build157286122/b001/exe/autocomplete
And I know this is when invoking the top.Largest since I've removed the piece
of code and then the job runs fine, could you please point me in the right
direction why my local func is not encoable as a interface {} and I will of
course happily send a PR when this is working on direct and the dataflow direct
so I can move on to the other examples
(All changes can be seen here)
[https://github.com/apache/beam/compare/master...ptomasroos:make-autocomplete-dataflowable?expand=1]
https://github.com/apache/beam/blob/master/sdks/go/examples/complete/autocomplete/autocomplete.go#L63
2018/09/10 13:37:10 Running autocomplete
Unable to encode combiner for lifting: failed to encode custom coder: bad
underlying type: bad field type: bad element: unencodable type: interface
{}2018/09/10 13:37:10 Using running binary as worker binary:
'/tmp/go-build157286122/b001/exe/autocomplete'
2018/09/10 13:37:10 Staging worker binary:
/tmp/go-build157286122/b001/exe/autocomplete
> Running autocomplete.go on dataflow fails
> -----------------------------------------
>
> Key: BEAM-5350
> URL: https://issues.apache.org/jira/browse/BEAM-5350
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Affects Versions: Not applicable
> Reporter: Tomas Roos
> Assignee: Henning Rohde
> Priority: Major
>
> I'm in the process as a external developer make sure that all examples are
> runnable on both direct and the dataflow runner as its crucial for people
> onboarding this project.
>
> I've visted the projects before and some are runnable, some probably where
> previously, and some are def not runnable.
>
> So I started top down today, in order to make autocomplete.go run on dataflow
> as well as the direct runner i changed the input in order to make it platform
> independent instead of pointing to a local file.The reading of the source
> from the public cloud storage went fine but it fails to run the top.Largest
> anonymous less function (ran on id: go-job-1-1536575613531078735) failed with
>
>
> {code:java}
> RESP: instruction_id: "-205" error: "Invalid bundle desc: decode: bad userfn:
> bad struct encoding: failed to decode data: decode: failed to find symbol
> main.main.func1: main.main.func1 not found. Use runtime.RegisterFunction in
> unit tests" register: < >
>
> {code}
>
> [https://github.com/apache/beam/blob/master/sdks/go/examples/complete/autocomplete/autocomplete.go#L63]
>
> So in order to fix this I introduced the local func called lessFn and
> registered in the init process. This though now instead when running
>
> {code:java}
>
> go run autocomplete.go --project fair-app-213019 --runner dataflow
> --staging_location=gs://fair-app-213019/staging-test2
> --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
> {code}
>
>
> fails with
>
> {code:java}
> 2018/09/10 13:37:10 Running autocomplete
> Unable to encode combiner for lifting: failed to encode custom coder: bad
> underlying type: bad field type: bad element: unencodable type: interface
> {}2018/09/10 13:37:10 Using running binary as worker binary:
> '/tmp/go-build157286122/b001/exe/autocomplete'
> 2018/09/10 13:37:10 Staging worker binary:
> /tmp/go-build157286122/b001/exe/autocomplete{code}
>
> And I know this is when invoking the top.Largest since I've removed the piece
> of code and then the job runs fine, could you please point me in the right
> direction why my local func is not encoable as a interface {} and I will of
> course happily send a PR when this is working on direct and the dataflow
> direct so I can move on to the other examples
>
> (All changes can be seen here)
> [https://github.com/apache/beam/compare/master...ptomasroos:make-autocomplete-dataflowable?expand=1]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
