[jira] [Commented] (BEAM-5350) Running autocomplete.go on dataflow fails
[ https://issues.apache.org/jira/browse/BEAM-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613297#comment-16613297 ] Tomas Roos commented on BEAM-5350: -- Thanks for the pointer! I will take a look if I can figure out what could be needed in order to resolve 4472. > Running autocomplete.go on dataflow fails > - > > Key: BEAM-5350 > URL: https://issues.apache.org/jira/browse/BEAM-5350 > Project: Beam > Issue Type: Bug > Components: sdk-go >Affects Versions: Not applicable >Reporter: Tomas Roos >Assignee: Henning Rohde >Priority: Major > > I'm in the process as a external developer make sure that all examples are > runnable on both direct and the dataflow runner as its crucial for people > onboarding this project. > > I've visted the projects before and some are runnable, some probably where > previously, and some are def not runnable. > > So I started top down today, in order to make autocomplete.go run on dataflow > as well as the direct runner i changed the input in order to make it platform > independent instead of pointing to a local file.The reading of the source > from the public cloud storage went fine but it fails to run the top.Largest > anonymous less function (ran on id: go-job-1-1536575613531078735) failed with > > > {code:java} > RESP: instruction_id: "-205" error: "Invalid bundle desc: decode: bad userfn: > bad struct encoding: failed to decode data: decode: failed to find symbol > main.main.func1: main.main.func1 not found. Use runtime.RegisterFunction in > unit tests" register: < > > > {code} > > [https://github.com/apache/beam/blob/master/sdks/go/examples/complete/autocomplete/autocomplete.go#L63] > > So in order to fix this I introduced the local func called lessFn and > registered in the init process. This though now instead when running > > {code:java} > > go run autocomplete.go --project fair-app-213019 --runner dataflow > --staging_location=gs://fair-app-213019/staging-test2 > --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515 > {code} > > > fails with > > {code:java} > 2018/09/10 13:37:10 Running autocomplete > Unable to encode combiner for lifting: failed to encode custom coder: bad > underlying type: bad field type: bad element: unencodable type: interface > {}2018/09/10 13:37:10 Using running binary as worker binary: > '/tmp/go-build157286122/b001/exe/autocomplete' > 2018/09/10 13:37:10 Staging worker binary: > /tmp/go-build157286122/b001/exe/autocomplete{code} > > And I know this is when invoking the top.Largest since I've removed the piece > of code and then the job runs fine, could you please point me in the right > direction why my local func is not encoable as a interface {} and I will of > course happily send a PR when this is working on direct and the dataflow > direct so I can move on to the other examples > > (All changes can be seen here) > [https://github.com/apache/beam/compare/master...ptomasroos:make-autocomplete-dataflowable?expand=1] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5350) Running autocomplete.go on dataflow fails
[ https://issues.apache.org/jira/browse/BEAM-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610990#comment-16610990 ] Robert Burke commented on BEAM-5350: BEAM-4472(https://issues.apache.org/jira/browse/BEAM-4472) is about the Top issue. The accumulator type is not presently encodeable and it wasn't used at all until Combiner Lifting was properly added. A good avenue to fixing that would be to do something similar to beam.Create for handling encoding those values instead of what's currently written. > Running autocomplete.go on dataflow fails > - > > Key: BEAM-5350 > URL: https://issues.apache.org/jira/browse/BEAM-5350 > Project: Beam > Issue Type: Bug > Components: sdk-go >Affects Versions: Not applicable >Reporter: Tomas Roos >Assignee: Henning Rohde >Priority: Major > > I'm in the process as a external developer make sure that all examples are > runnable on both direct and the dataflow runner as its crucial for people > onboarding this project. > > I've visted the projects before and some are runnable, some probably where > previously, and some are def not runnable. > > So I started top down today, in order to make autocomplete.go run on dataflow > as well as the direct runner i changed the input in order to make it platform > independent instead of pointing to a local file.The reading of the source > from the public cloud storage went fine but it fails to run the top.Largest > anonymous less function (ran on id: go-job-1-1536575613531078735) failed with > > > {code:java} > RESP: instruction_id: "-205" error: "Invalid bundle desc: decode: bad userfn: > bad struct encoding: failed to decode data: decode: failed to find symbol > main.main.func1: main.main.func1 not found. Use runtime.RegisterFunction in > unit tests" register: < > > > {code} > > [https://github.com/apache/beam/blob/master/sdks/go/examples/complete/autocomplete/autocomplete.go#L63] > > So in order to fix this I introduced the local func called lessFn and > registered in the init process. This though now instead when running > > {code:java} > > go run autocomplete.go --project fair-app-213019 --runner dataflow > --staging_location=gs://fair-app-213019/staging-test2 > --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515 > {code} > > > fails with > > {code:java} > 2018/09/10 13:37:10 Running autocomplete > Unable to encode combiner for lifting: failed to encode custom coder: bad > underlying type: bad field type: bad element: unencodable type: interface > {}2018/09/10 13:37:10 Using running binary as worker binary: > '/tmp/go-build157286122/b001/exe/autocomplete' > 2018/09/10 13:37:10 Staging worker binary: > /tmp/go-build157286122/b001/exe/autocomplete{code} > > And I know this is when invoking the top.Largest since I've removed the piece > of code and then the job runs fine, could you please point me in the right > direction why my local func is not encoable as a interface {} and I will of > course happily send a PR when this is working on direct and the dataflow > direct so I can move on to the other examples > > (All changes can be seen here) > [https://github.com/apache/beam/compare/master...ptomasroos:make-autocomplete-dataflowable?expand=1] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5350) Running autocomplete.go on dataflow fails
[ https://issues.apache.org/jira/browse/BEAM-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610110#comment-16610110 ] Tomas Roos commented on BEAM-5350: -- I will try to complete it according to the java example, my idea was to get the current running before moving on to updating it :) Will send a PR hopefully later today > Running autocomplete.go on dataflow fails > - > > Key: BEAM-5350 > URL: https://issues.apache.org/jira/browse/BEAM-5350 > Project: Beam > Issue Type: Bug > Components: sdk-go >Affects Versions: Not applicable >Reporter: Tomas Roos >Assignee: Henning Rohde >Priority: Major > > I'm in the process as a external developer make sure that all examples are > runnable on both direct and the dataflow runner as its crucial for people > onboarding this project. > > I've visted the projects before and some are runnable, some probably where > previously, and some are def not runnable. > > So I started top down today, in order to make autocomplete.go run on dataflow > as well as the direct runner i changed the input in order to make it platform > independent instead of pointing to a local file.The reading of the source > from the public cloud storage went fine but it fails to run the top.Largest > anonymous less function (ran on id: go-job-1-1536575613531078735) failed with > > > {code:java} > RESP: instruction_id: "-205" error: "Invalid bundle desc: decode: bad userfn: > bad struct encoding: failed to decode data: decode: failed to find symbol > main.main.func1: main.main.func1 not found. Use runtime.RegisterFunction in > unit tests" register: < > > > {code} > > [https://github.com/apache/beam/blob/master/sdks/go/examples/complete/autocomplete/autocomplete.go#L63] > > So in order to fix this I introduced the local func called lessFn and > registered in the init process. This though now instead when running > > {code:java} > > go run autocomplete.go --project fair-app-213019 --runner dataflow > --staging_location=gs://fair-app-213019/staging-test2 > --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515 > {code} > > > fails with > > {code:java} > 2018/09/10 13:37:10 Running autocomplete > Unable to encode combiner for lifting: failed to encode custom coder: bad > underlying type: bad field type: bad element: unencodable type: interface > {}2018/09/10 13:37:10 Using running binary as worker binary: > '/tmp/go-build157286122/b001/exe/autocomplete' > 2018/09/10 13:37:10 Staging worker binary: > /tmp/go-build157286122/b001/exe/autocomplete{code} > > And I know this is when invoking the top.Largest since I've removed the piece > of code and then the job runs fine, could you please point me in the right > direction why my local func is not encoable as a interface {} and I will of > course happily send a PR when this is working on direct and the dataflow > direct so I can move on to the other examples > > (All changes can be seen here) > [https://github.com/apache/beam/compare/master...ptomasroos:make-autocomplete-dataflowable?expand=1] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5350) Running autocomplete.go on dataflow fails
[ https://issues.apache.org/jira/browse/BEAM-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609984#comment-16609984 ] Henning Rohde commented on BEAM-5350: - Tomas -- do you want to re-purpose this JIRA to remove (or complete) the example? If you send a PR to either effect, I can merge it to avoid a defunct example. For some reason, I can't simply assign you the JIRA. > Running autocomplete.go on dataflow fails > - > > Key: BEAM-5350 > URL: https://issues.apache.org/jira/browse/BEAM-5350 > Project: Beam > Issue Type: Bug > Components: sdk-go >Affects Versions: Not applicable >Reporter: Tomas Roos >Assignee: Henning Rohde >Priority: Major > > I'm in the process as a external developer make sure that all examples are > runnable on both direct and the dataflow runner as its crucial for people > onboarding this project. > > I've visted the projects before and some are runnable, some probably where > previously, and some are def not runnable. > > So I started top down today, in order to make autocomplete.go run on dataflow > as well as the direct runner i changed the input in order to make it platform > independent instead of pointing to a local file.The reading of the source > from the public cloud storage went fine but it fails to run the top.Largest > anonymous less function (ran on id: go-job-1-1536575613531078735) failed with > > > {code:java} > RESP: instruction_id: "-205" error: "Invalid bundle desc: decode: bad userfn: > bad struct encoding: failed to decode data: decode: failed to find symbol > main.main.func1: main.main.func1 not found. Use runtime.RegisterFunction in > unit tests" register: < > > > {code} > > [https://github.com/apache/beam/blob/master/sdks/go/examples/complete/autocomplete/autocomplete.go#L63] > > So in order to fix this I introduced the local func called lessFn and > registered in the init process. This though now instead when running > > {code:java} > > go run autocomplete.go --project fair-app-213019 --runner dataflow > --staging_location=gs://fair-app-213019/staging-test2 > --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515 > {code} > > > fails with > > {code:java} > 2018/09/10 13:37:10 Running autocomplete > Unable to encode combiner for lifting: failed to encode custom coder: bad > underlying type: bad field type: bad element: unencodable type: interface > {}2018/09/10 13:37:10 Using running binary as worker binary: > '/tmp/go-build157286122/b001/exe/autocomplete' > 2018/09/10 13:37:10 Staging worker binary: > /tmp/go-build157286122/b001/exe/autocomplete{code} > > And I know this is when invoking the top.Largest since I've removed the piece > of code and then the job runs fine, could you please point me in the right > direction why my local func is not encoable as a interface {} and I will of > course happily send a PR when this is working on direct and the dataflow > direct so I can move on to the other examples > > (All changes can be seen here) > [https://github.com/apache/beam/compare/master...ptomasroos:make-autocomplete-dataflowable?expand=1] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5350) Running autocomplete.go on dataflow fails
[ https://issues.apache.org/jira/browse/BEAM-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609950#comment-16609950 ] Henning Rohde commented on BEAM-5350: - That is a really awesome idea. Thanks for taking that on! Let's consolidate the investigation in https://issues.apache.org/jira/browse/BEAM-5253. > Running autocomplete.go on dataflow fails > - > > Key: BEAM-5350 > URL: https://issues.apache.org/jira/browse/BEAM-5350 > Project: Beam > Issue Type: Bug > Components: sdk-go >Affects Versions: Not applicable >Reporter: Tomas Roos >Assignee: Henning Rohde >Priority: Major > > I'm in the process as a external developer make sure that all examples are > runnable on both direct and the dataflow runner as its crucial for people > onboarding this project. > > I've visted the projects before and some are runnable, some probably where > previously, and some are def not runnable. > > So I started top down today, in order to make autocomplete.go run on dataflow > as well as the direct runner i changed the input in order to make it platform > independent instead of pointing to a local file.The reading of the source > from the public cloud storage went fine but it fails to run the top.Largest > anonymous less function (ran on id: go-job-1-1536575613531078735) failed with > > > {code:java} > RESP: instruction_id: "-205" error: "Invalid bundle desc: decode: bad userfn: > bad struct encoding: failed to decode data: decode: failed to find symbol > main.main.func1: main.main.func1 not found. Use runtime.RegisterFunction in > unit tests" register: < > > > {code} > > [https://github.com/apache/beam/blob/master/sdks/go/examples/complete/autocomplete/autocomplete.go#L63] > > So in order to fix this I introduced the local func called lessFn and > registered in the init process. This though now instead when running > > {code:java} > > go run autocomplete.go --project fair-app-213019 --runner dataflow > --staging_location=gs://fair-app-213019/staging-test2 > --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515 > {code} > > > fails with > > {code:java} > 2018/09/10 13:37:10 Running autocomplete > Unable to encode combiner for lifting: failed to encode custom coder: bad > underlying type: bad field type: bad element: unencodable type: interface > {}2018/09/10 13:37:10 Using running binary as worker binary: > '/tmp/go-build157286122/b001/exe/autocomplete' > 2018/09/10 13:37:10 Staging worker binary: > /tmp/go-build157286122/b001/exe/autocomplete{code} > > And I know this is when invoking the top.Largest since I've removed the piece > of code and then the job runs fine, could you please point me in the right > direction why my local func is not encoable as a interface {} and I will of > course happily send a PR when this is working on direct and the dataflow > direct so I can move on to the other examples > > (All changes can be seen here) > [https://github.com/apache/beam/compare/master...ptomasroos:make-autocomplete-dataflowable?expand=1] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5350) Running autocomplete.go on dataflow fails
[ https://issues.apache.org/jira/browse/BEAM-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609622#comment-16609622 ] Henning Rohde commented on BEAM-5350: - The Go autocomplete example is a stub from a long time ago. It was never completed and it makes sense to either complete (so that it computes the same thing as the Java example) or delete. > Running autocomplete.go on dataflow fails > - > > Key: BEAM-5350 > URL: https://issues.apache.org/jira/browse/BEAM-5350 > Project: Beam > Issue Type: Bug > Components: sdk-go >Affects Versions: Not applicable >Reporter: Tomas Roos >Assignee: Henning Rohde >Priority: Major > > I'm in the process as a external developer make sure that all examples are > runnable on both direct and the dataflow runner as its crucial for people > onboarding this project. > > I've visted the projects before and some are runnable, some probably where > previously, and some are def not runnable. > > So I started top down today, in order to make autocomplete.go run on dataflow > as well as the direct runner i changed the input in order to make it platform > independent instead of pointing to a local file.The reading of the source > from the public cloud storage went fine but it fails to run the top.Largest > anonymous less function (ran on id: go-job-1-1536575613531078735) failed with > > > {code:java} > RESP: instruction_id: "-205" error: "Invalid bundle desc: decode: bad userfn: > bad struct encoding: failed to decode data: decode: failed to find symbol > main.main.func1: main.main.func1 not found. Use runtime.RegisterFunction in > unit tests" register: < > > > {code} > > [https://github.com/apache/beam/blob/master/sdks/go/examples/complete/autocomplete/autocomplete.go#L63] > > So in order to fix this I introduced the local func called lessFn and > registered in the init process. This though now instead when running > > {code:java} > > go run autocomplete.go --project fair-app-213019 --runner dataflow > --staging_location=gs://fair-app-213019/staging-test2 > --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515 > {code} > > > fails with > > {code:java} > 2018/09/10 13:37:10 Running autocomplete > Unable to encode combiner for lifting: failed to encode custom coder: bad > underlying type: bad field type: bad element: unencodable type: interface > {}2018/09/10 13:37:10 Using running binary as worker binary: > '/tmp/go-build157286122/b001/exe/autocomplete' > 2018/09/10 13:37:10 Staging worker binary: > /tmp/go-build157286122/b001/exe/autocomplete{code} > > And I know this is when invoking the top.Largest since I've removed the piece > of code and then the job runs fine, could you please point me in the right > direction why my local func is not encoable as a interface {} and I will of > course happily send a PR when this is working on direct and the dataflow > direct so I can move on to the other examples > > (All changes can be seen here) > [https://github.com/apache/beam/compare/master...ptomasroos:make-autocomplete-dataflowable?expand=1] -- This message was sent by Atlassian JIRA (v7.6.3#76005)