lostluck commented on a change in pull request #15447: URL: https://github.com/apache/beam/pull/15447#discussion_r703902787
########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6416,6 +6421,68 @@ with pipeline as p: 4. After the job has been submitted to the Beam runner, shutdown the expansion service by terminating the expansion service process. +#### 13.2.3. Using cross-language transforms in a Go pipeline + +If a Go-specific wrapper for a cross-language is available, use that; otherwise, you have to use the +lower-level [CrossLanguage](https://pkg.go.dev/github.com/apache/beam/sdks/go/pkg/beam#CrossLanguage) +function to access the transform. + +**Expansion Services** + +The Go SDK does not yet support automatically starting an expansion service. In order to use +cross-language transforms, you must manually start any necessary expansion services on your local +machine and ensure they are accessible to your code during pipeline construction. + +**Using an SDK wrapper** + +To use a cross-language transform through an SDK wrapper, include the module for the SDK wrapper +and call it from your pipeline as shown in the example: + +{{< highlight >}} +import ( + "github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/kafkaio" +) + +// Kafka Read using previously defined values. +kafkaRecords := kafkaio.Read( + s, + expansionAddr, // Address of expansion service. + bootstrapAddr, + []string{topicName}, + kafkaio.MaxNumRecords(numRecords), + kafkaio.ConsumerConfigs(map[string]string{"auto.offset.reset": "earliest"})) +{{< /highlight >}} + +**Using the CrossLanguage function** + +When an SDK-specific wrapper isn't available, you will have to access the cross-language transform through the `CrossLanguage` function. + +1. Make sure you have the appropriate expansion service running. See the expansion service section for details. +2. Make sure the transform you're trying to use is available and can be used by the expansion service. + For Java, make sure the builder and registrar for the transform are available in the classpath of + the expansion service. +3. Use the `CrossLanguage` function in your pipeline as appropriate. Reference the URN, Payload, + expansion service address, and define inputs and outputs. You can use the + [CrossLanguagePayload](https://pkg.go.dev/github.com/apache/beam/sdks/go/pkg/beam#CrossLanguagePayload) + function as a helper for encoding a payload. You can use the + [UnnamedInput](https://pkg.go.dev/github.com/apache/beam/sdks/go/pkg/beam#UnnamedInput) and + [UnnamedOutput](https://pkg.go.dev/github.com/apache/beam/sdks/go/pkg/beam#UnnamedOutput) + functions as shortcuts for single, unnamed inputs/outputs or define a map for named ones. + + {{< highlight >}} +type prefixPayload struct { + Data string +} +urn := "beam:transforms:xlang:test:prefix" +payload := beam.CrossLanguagePayload(prefixPayload{Data: prefix}) +expansionAddr := <Address of expansion service> +outT := beam.UnnamedOutput(typex.New(reflectx.String)) +res := beam.CrossLanguage(s, urn, payload, expansionAddr, beam.UnnamedInput(inputPCol)) Review comment: Missing use of the outT parameter. This is where the various snippets of code samples are handy. Avoiding writing code that can't compile, since we are executing it (or at least building it) in our test suites! ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6350,6 +6350,11 @@ $ python -m apache_beam.runners.portability.expansion_service_test -p $PORT_FOR_ Currently Python external transforms are limited to dependencies available in core Beam SDK Harness. +#### 13.1.3. Creating cross-language Go transforms + +Go currently does not support creating cross-language transforms, only using cross-language +transforms from other languages. Review comment: Consider adding a JIRA task reference to this text for folks to track/contribute/refer to. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6416,6 +6421,68 @@ with pipeline as p: 4. After the job has been submitted to the Beam runner, shutdown the expansion service by terminating the expansion service process. +#### 13.2.3. Using cross-language transforms in a Go pipeline + +If a Go-specific wrapper for a cross-language is available, use that; otherwise, you have to use the +lower-level [CrossLanguage](https://pkg.go.dev/github.com/apache/beam/sdks/go/pkg/beam#CrossLanguage) +function to access the transform. + +**Expansion Services** + +The Go SDK does not yet support automatically starting an expansion service. In order to use +cross-language transforms, you must manually start any necessary expansion services on your local +machine and ensure they are accessible to your code during pipeline construction. + +**Using an SDK wrapper** + +To use a cross-language transform through an SDK wrapper, include the module for the SDK wrapper +and call it from your pipeline as shown in the example: + +{{< highlight >}} +import ( + "github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/kafkaio" +) + +// Kafka Read using previously defined values. +kafkaRecords := kafkaio.Read( + s, + expansionAddr, // Address of expansion service. + bootstrapAddr, + []string{topicName}, + kafkaio.MaxNumRecords(numRecords), + kafkaio.ConsumerConfigs(map[string]string{"auto.offset.reset": "earliest"})) +{{< /highlight >}} + +**Using the CrossLanguage function** + +When an SDK-specific wrapper isn't available, you will have to access the cross-language transform through the `CrossLanguage` function. + +1. Make sure you have the appropriate expansion service running. See the expansion service section for details. +2. Make sure the transform you're trying to use is available and can be used by the expansion service. + For Java, make sure the builder and registrar for the transform are available in the classpath of Review comment: Consider referring to the Java and Python sections above about how to make transforms available via their expansion services. Given the active proposal to simplify the java work, we don't want to be too specific about how they work in the Go section. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6416,6 +6421,68 @@ with pipeline as p: 4. After the job has been submitted to the Beam runner, shutdown the expansion service by terminating the expansion service process. +#### 13.2.3. Using cross-language transforms in a Go pipeline + +If a Go-specific wrapper for a cross-language is available, use that; otherwise, you have to use the +lower-level [CrossLanguage](https://pkg.go.dev/github.com/apache/beam/sdks/go/pkg/beam#CrossLanguage) +function to access the transform. + +**Expansion Services** + +The Go SDK does not yet support automatically starting an expansion service. In order to use +cross-language transforms, you must manually start any necessary expansion services on your local +machine and ensure they are accessible to your code during pipeline construction. Review comment: I'm on the fence about adding a JIRA for folks to reference to contribute this feature. As you like. Probably better after we add the Xlang override/registration if needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
