lostluck commented on a change in pull request #15447: URL: https://github.com/apache/beam/pull/15447#discussion_r705801413
########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6416,6 +6421,68 @@ with pipeline as p: 4. After the job has been submitted to the Beam runner, shutdown the expansion service by terminating the expansion service process. +#### 13.2.3. Using cross-language transforms in a Go pipeline + +If a Go-specific wrapper for a cross-language is available, use that; otherwise, you have to use the +lower-level [CrossLanguage](https://pkg.go.dev/github.com/apache/beam/sdks/go/pkg/beam#CrossLanguage) +function to access the transform. + +**Expansion Services** + +The Go SDK does not yet support automatically starting an expansion service. In order to use +cross-language transforms, you must manually start any necessary expansion services on your local +machine and ensure they are accessible to your code during pipeline construction; see more at +[BEAM-12862](https://issues.apache.org/jira/browse/BEAM-12862). + +**Using an SDK wrapper** + +To use a cross-language transform through an SDK wrapper, include the module for the SDK wrapper +and call it from your pipeline as shown in the example: + +{{< highlight >}} +import ( + "github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/kafkaio" +) + +// Kafka Read using previously defined values. +kafkaRecords := kafkaio.Read( + s, + expansionAddr, // Address of expansion service. + bootstrapAddr, + []string{topicName}, + kafkaio.MaxNumRecords(numRecords), + kafkaio.ConsumerConfigs(map[string]string{"auto.offset.reset": "earliest"})) +{{< /highlight >}} + +**Using the CrossLanguage function** + +When an SDK-specific wrapper isn't available, you will have to access the cross-language transform through the `CrossLanguage` function. Review comment: ```suggestion When an SDK-specific wrapper isn't available, you will have to access the cross-language transform through the `beam.CrossLanguage` function. ``` Go functions are package namespaced so it's more natural in external documentation to refer to them that way. (Doesn't practically apply to GoDoc, but this isn't GoDoc.) Here and below. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6416,6 +6421,68 @@ with pipeline as p: 4. After the job has been submitted to the Beam runner, shutdown the expansion service by terminating the expansion service process. +#### 13.2.3. Using cross-language transforms in a Go pipeline + +If a Go-specific wrapper for a cross-language is available, use that; otherwise, you have to use the +lower-level [CrossLanguage](https://pkg.go.dev/github.com/apache/beam/sdks/go/pkg/beam#CrossLanguage) +function to access the transform. + +**Expansion Services** + +The Go SDK does not yet support automatically starting an expansion service. In order to use +cross-language transforms, you must manually start any necessary expansion services on your local +machine and ensure they are accessible to your code during pipeline construction; see more at +[BEAM-12862](https://issues.apache.org/jira/browse/BEAM-12862). + +**Using an SDK wrapper** + +To use a cross-language transform through an SDK wrapper, include the module for the SDK wrapper Review comment: I'd say "import the package" since that's the go term for this. In this case we're communicating with Go programmers. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6416,6 +6421,68 @@ with pipeline as p: 4. After the job has been submitted to the Beam runner, shutdown the expansion service by terminating the expansion service process. +#### 13.2.3. Using cross-language transforms in a Go pipeline + +If a Go-specific wrapper for a cross-language is available, use that; otherwise, you have to use the +lower-level [CrossLanguage](https://pkg.go.dev/github.com/apache/beam/sdks/go/pkg/beam#CrossLanguage) +function to access the transform. + +**Expansion Services** + +The Go SDK does not yet support automatically starting an expansion service. In order to use +cross-language transforms, you must manually start any necessary expansion services on your local +machine and ensure they are accessible to your code during pipeline construction; see more at +[BEAM-12862](https://issues.apache.org/jira/browse/BEAM-12862). + +**Using an SDK wrapper** + +To use a cross-language transform through an SDK wrapper, include the module for the SDK wrapper +and call it from your pipeline as shown in the example: + +{{< highlight >}} +import ( + "github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/kafkaio" +) + +// Kafka Read using previously defined values. +kafkaRecords := kafkaio.Read( + s, + expansionAddr, // Address of expansion service. + bootstrapAddr, + []string{topicName}, + kafkaio.MaxNumRecords(numRecords), + kafkaio.ConsumerConfigs(map[string]string{"auto.offset.reset": "earliest"})) +{{< /highlight >}} + +**Using the CrossLanguage function** + +When an SDK-specific wrapper isn't available, you will have to access the cross-language transform through the `CrossLanguage` function. + +1. Make sure you have the appropriate expansion service running. See the expansion service section for details. +2. Make sure the transform you're trying to use is available and can be used by the expansion service. + Refer to [Creating cross-language transforms](#create-x-lang-transforms) for details. +3. Use the `CrossLanguage` function in your pipeline as appropriate. Reference the URN, Payload, + expansion service address, and define inputs and outputs. You can use the + [CrossLanguagePayload](https://pkg.go.dev/github.com/apache/beam/sdks/go/pkg/beam#CrossLanguagePayload) + function as a helper for encoding a payload. You can use the + [UnnamedInput](https://pkg.go.dev/github.com/apache/beam/sdks/go/pkg/beam#UnnamedInput) and + [UnnamedOutput](https://pkg.go.dev/github.com/apache/beam/sdks/go/pkg/beam#UnnamedOutput) + functions as shortcuts for single, unnamed inputs/outputs or define a map for named ones. + + {{< highlight >}} +type prefixPayload struct { + Data string Review comment: ```suggestion Data string `beam:"data"` ``` Minor nit: Probably better to include the lowercasing/for schema purposes. We never send this schema over, and it will encode identically, but may as well get into the habit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
