[
https://issues.apache.org/jira/browse/BEAM-11574?focusedWorklogId=576388&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-576388
]
ASF GitHub Bot logged work on BEAM-11574:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Apr/21 04:22
Start Date: 03/Apr/21 04:22
Worklog Time Spent: 10m
Work Description: youngoli commented on a change in pull request #14397:
URL: https://github.com/apache/beam/pull/14397#discussion_r606615602
##########
File path: sdks/go/pkg/beam/core/runtime/pipelinex/replace.go
##########
@@ -243,3 +245,33 @@ func findFreeName(seen map[string]bool, name string)
string {
}
}
}
+
+// ApplySdkImageOverrides takes a pipeline and a map of patterns to overrides,
+// and proceeds to replace matching ContainerImages in any Environments
+// present in the pipeline.
+func ApplySdkImageOverrides(p *pipepb.Pipeline, patterns map[string]string)
error {
Review comment:
Done. Also adding this to the flag description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 576388)
Time Spent: 1h 20m (was: 1h 10m)
> Enable x-lang on Dataflow side for integration tests
> ----------------------------------------------------
>
> Key: BEAM-11574
> URL: https://issues.apache.org/jira/browse/BEAM-11574
> Project: Beam
> Issue Type: Bug
> Components: cross-language, sdk-go
> Reporter: Daniel Oliveira
> Assignee: Daniel Oliveira
> Priority: P2
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Dataflow x-lang ValidatesRunner tests are failing with the following error:
> {noformat}
> panic: unmarshalling coder UIbLsVLhrXKvCoder
> unmarshalling coder UIbLsVLhrXVoidCoder
> could not unmarshal coder from spec:{urn:"beam:coders:javasdk:0.1"
> payload:"\x82SNAPPY\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x88\xd6\x01\xe8\xac\xed\x00\x05sr\x00$org.apache.beam.sdk.coders.VoidCoder\xb9\xbfU\x9b\xe8\r\xafU\x02\x00\x00xr\x00&j3\x00\x14Atomic\x055
>
> \xc7\xec\xb5̅tPF\x02\x055\x00*j5\x00$Structured\x059\x1cs\xbf\x12\x0e\xd5\xd46\x11\t9\x00
> j9\x00\x05/0C\xddՉ\xae\xbc~\xf8\x02\x00\x00xp"}, unknown URN
> beam:coders:javasdk:0.1 [recovered]
> panic: unmarshalling coder UIbLsVLhrXKvCoder
> unmarshalling coder UIbLsVLhrXVoidCoder
> could not unmarshal coder from spec:{urn:"beam:coders:javasdk:0.1"
> payload:"\x82SNAPPY\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x88\xd6\x01\xe8\xac\xed\x00\x05sr\x00$org.apache.beam.sdk.coders.VoidCoder\xb9\xbfU\x9b\xe8\r\xafU\x02\x00\x00xr\x00&j3\x00\x14Atomic\x055
>
> \xc7\xec\xb5̅tPF\x02\x055\x00*j5\x00$Structured\x059\x1cs\xbf\x12\x0e\xd5\xd46\x11\t9\x00
> j9\x00\x05/0C\xddՉ\xae\xbc~\xf8\x02\x00\x00xp"}, unknown URN
> beam:coders:javasdk:0.1
> goroutine 130 [running]:
> testing.tRunner.func1.1(0xe2f1a0, 0xc00059ea80)
> /usr/lib/google-golang/src/testing/testing.go:1072 +0x30d
> testing.tRunner.func1(0xc000290f00)
> /usr/lib/google-golang/src/testing/testing.go:1075 +0x41a
> panic(0xe2f1a0, 0xc00059ea80)
> /usr/lib/google-golang/src/runtime/panic.go:969 +0x1b9
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib.(*translator).translateCoder(0xc0002da260,
> 0xc0005b5b00, 0xc0003fdee0, 0x11, 0xc000379ac8)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go:337
> +0xa5
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib.(*translator).translateOutputs(0xc0002da260,
> 0xc00079b1d0, 0xc0003fd600, 0x11, 0xc000376180)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go:317
> +0x190
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib.(*translator).translateTransform(0xc0002da260,
> 0xc00019e330, 0x2d, 0xc00078a660, 0x1b, 0xf15a80, 0xc000096000,
> 0xc0004fb898, 0x6475eb, 0xc000096000)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go:111
> +0x111
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib.(*translator).translateTransforms(0xc0002da260,
> 0xc00019e330, 0x2d, 0xc000265580, 0x7, 0x8, 0x2d, 0x6, 0x0, 0xc00003f41a,
> ...)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go:97
> +0xda
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib.(*translator).translateTransform(0xc0002da260,
> 0xc000280600, 0x1a, 0xc0001266d8, 0x2, 0xf15a80, 0xc000096000, 0xc0004fc710,
> 0x6475eb, 0xc000096000)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go:285
> +0x36d
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib.(*translator).translateTransforms(0xc0002da260,
> 0xc000280600, 0x1a, 0xc0002be740, 0x1, 0x1, 0x1a, 0x1, 0x0, 0x0, ...)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go:97
> +0xda
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib.(*translator).translateTransform(0xc0002da260,
> 0x0, 0x0, 0xc0001266d4, 0x2, 0x1, 0x1, 0x2, 0x0, 0x0)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go:285
> +0x36d
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib.(*translator).translateTransforms(0xc0002da260,
> 0x0, 0x0, 0xc000180140, 0x5, 0x5, 0xc0000a15f8, 0x5b96a5, 0xc000102600,
> 0x200000003, ...)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go:97
> +0xda
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib.translate(0xc000313800,
> 0x5d4fda, 0x1, 0x2, 0xc000778000, 0xa2)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go:73
> +0x77
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib.Translate(0x103ff00,
> 0xc00019c628, 0xc000313800, 0xc0002bbcb0, 0xc00019b900, 0x77, 0xc00016e2d0,
> 0x84, 0xc00019b880, 0x76, ...)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go:77
> +0x45
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib.Execute(0x103ff00,
> 0xc00019c628, 0xc000313800, 0xc0002bbcb0, 0xc00019b900, 0x77, 0xc00016e2d0,
> 0x84, 0xc00019b880, 0x76, ...)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflowlib/execute.go:91
> +0x699
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow.Execute(0x103ff00,
> 0xc00019c628, 0xc000124888, 0x8, 0xc000142158, 0xf6ae01, 0xc00032ba40)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runners/dataflow/dataflow.go:207
> +0xe7d
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam.Run(0x103ff00,
> 0xc00019c628, 0x7ffec804dad3, 0x8, 0xc000124888, 0x102d860, 0xc00032b980,
> 0xc0002b7ee8, 0xba6e45)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/runner.go:50
> +0x87
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/testing/ptest.Run(0xc000124888,
> 0xc00032a660, 0xc00079be00)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/testing/ptest/ptest.go:89
> +0x8b
> github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/testing/ptest.RunAndValidate(0xc000290f00,
> 0xc000124888)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/testing/ptest/ptest.go:96
> +0x2f
> github.com/apache/beam/sdks/go/test/integration/xlang.TestXLang_CombineGlobally(0xc000290f00)
>
> /usr/local/google/home/danoliveira/go/src/github.com/apache/beam/sdks/go/test/integration/xlang/xlang_test.go:165
> +0x249
> testing.tRunner(0xc000290f00, 0xf6a908)
> /usr/lib/google-golang/src/testing/testing.go:1123 +0xef
> created by testing.(*T).Run
> /usr/lib/google-golang/src/testing/testing.go:1168 +0x2b3
> {noformat}
> This seems to imply that the bundles intended to be sent to the Java SDK are
> still being sent to the Go SDK (these are Go SDK errors), so it seems that
> cross-language functionality still needs to be enabled for the integration
> tests.
> Edit:
> Looking closer, the stacktrace actually suggests that the problem is that
> Go's Dataflow translation doesn't properly support cross-language transforms.
> The solution would be to adjust dataflowlib/translate.go to support
> cross-language.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
