chamikaramj commented on a change in pull request #15941: URL: https://github.com/apache/beam/pull/15941#discussion_r748653865
########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav #### 13.1.1. Creating cross-language Java transforms +There are two ways to make Java transforms available to other SDKs. + +* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code. +* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes. + +##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code + +Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example, +* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline +* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code Review comment: Done. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav #### 13.1.1. Creating cross-language Java transforms +There are two ways to make Java transforms available to other SDKs. + +* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code. +* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes. + +##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code + +Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example, +* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline +* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code + +> **Note:** This feature is currently only available when using Java transforms from a Python pipeline. + +To be eligible for direct usage, API of the Java transforms has to follow the following pattern. +* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class. Review comment: Done. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav #### 13.1.1. Creating cross-language Java transforms +There are two ways to make Java transforms available to other SDKs. + +* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code. +* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes. + +##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code + +Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example, +* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline +* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code + +> **Note:** This feature is currently only available when using Java transforms from a Python pipeline. + +To be eligible for direct usage, API of the Java transforms has to follow the following pattern. +* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class. +* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform. + +See below for the structure of an example Java class that can be directly used from the Python API. + +{{< highlight >}} +public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> { + . . . + + // Following method satisfies the Requirement 1. + // Note that you may also use a class constructor instead of a static method. + public static JavaDataGenerator create(Integer size) { + return new JavaDataGenerator(size); + } + + static class JavaDataGeneratorConfig implements Serializable { + public String prefix; + public long length; + public String suffix; + . . . + } + + // Following method conforms to the Requirement 2 + public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) { + return new JavaDataGenerator(this.size, javaDataGeneratorConfig); + } + + . . . +} +{{< /highlight >}} + +To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following. + +* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python. +* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value. +* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly + access Java transforms defined in the allowlist from the Python side. + +Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below. + +##### Step 1 + +To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names, +constructor methods, and builder methods that are directly available to be used from the Python side. + +Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which +denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API. + +{{< highlight >}} +version: v1 +allowedClasses: +- className: my.beam.transforms.JavaDataGenerator + allowedConstructorMethods: + - create + allowedBuilderMethods: + - withJavaDataGeneratorConfig +{{< /highlight >}} + +##### Step 2 + +The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started +as a local Java process using the following command. + +{{< highlight >}} +java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file> +{{< /highlight >}} + +Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies +if an expansion service address was not provided. + +##### Step 3 + +You can directly use the Java class from your Python pipeline using a stub transform created using the `JavaExternalTransform` API. This API allows you to construct the transform +using the Java class name and allows you to invoke builder methods to configure the class. + +Constructor and method parameter types are mapped between Python and Java using a Beam Schema. The Schema is auto-generated using the object types +provided in the Python side. If the Java class constructor method or builder method accepts any complex object types, make sure that Beam Schema Review comment: Done. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav #### 13.1.1. Creating cross-language Java transforms +There are two ways to make Java transforms available to other SDKs. + +* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code. +* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes. + +##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code + +Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example, +* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline +* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code + +> **Note:** This feature is currently only available when using Java transforms from a Python pipeline. + +To be eligible for direct usage, API of the Java transforms has to follow the following pattern. +* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class. +* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform. + +See below for the structure of an example Java class that can be directly used from the Python API. + +{{< highlight >}} +public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> { + . . . + + // Following method satisfies the Requirement 1. + // Note that you may also use a class constructor instead of a static method. + public static JavaDataGenerator create(Integer size) { + return new JavaDataGenerator(size); + } + + static class JavaDataGeneratorConfig implements Serializable { + public String prefix; + public long length; + public String suffix; + . . . + } + + // Following method conforms to the Requirement 2 + public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) { + return new JavaDataGenerator(this.size, javaDataGeneratorConfig); + } + + . . . +} +{{< /highlight >}} + +To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following. + +* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python. +* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value. +* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly + access Java transforms defined in the allowlist from the Python side. + +Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below. + +##### Step 1 + +To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names, +constructor methods, and builder methods that are directly available to be used from the Python side. + +Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which +denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API. + +{{< highlight >}} +version: v1 +allowedClasses: +- className: my.beam.transforms.JavaDataGenerator + allowedConstructorMethods: + - create + allowedBuilderMethods: + - withJavaDataGeneratorConfig +{{< /highlight >}} + +##### Step 2 + +The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started +as a local Java process using the following command. + +{{< highlight >}} +java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file> +{{< /highlight >}} + +Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies +if an expansion service address was not provided. + +##### Step 3 + +You can directly use the Java class from your Python pipeline using a stub transform created using the `JavaExternalTransform` API. This API allows you to construct the transform +using the Java class name and allows you to invoke builder methods to configure the class. + +Constructor and method parameter types are mapped between Python and Java using a Beam Schema. The Schema is auto-generated using the object types +provided in the Python side. If the Java class constructor method or builder method accepts any complex object types, make sure that Beam Schema +for these objects are registered and available for the Java expansion service. If a schema has not been registered, Java expansion service will Review comment: Done. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav #### 13.1.1. Creating cross-language Java transforms +There are two ways to make Java transforms available to other SDKs. + +* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code. +* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes. + +##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code + +Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example, +* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline +* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code + +> **Note:** This feature is currently only available when using Java transforms from a Python pipeline. + +To be eligible for direct usage, API of the Java transforms has to follow the following pattern. Review comment: Done. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav #### 13.1.1. Creating cross-language Java transforms +There are two ways to make Java transforms available to other SDKs. + +* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code. +* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes. + +##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code + +Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example, +* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline +* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code + +> **Note:** This feature is currently only available when using Java transforms from a Python pipeline. + +To be eligible for direct usage, API of the Java transforms has to follow the following pattern. +* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class. +* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform. + +See below for the structure of an example Java class that can be directly used from the Python API. + +{{< highlight >}} +public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> { + . . . + + // Following method satisfies the Requirement 1. + // Note that you may also use a class constructor instead of a static method. + public static JavaDataGenerator create(Integer size) { + return new JavaDataGenerator(size); + } + + static class JavaDataGeneratorConfig implements Serializable { + public String prefix; + public long length; + public String suffix; + . . . + } + + // Following method conforms to the Requirement 2 + public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) { + return new JavaDataGenerator(this.size, javaDataGeneratorConfig); + } + + . . . +} +{{< /highlight >}} + +To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following. + +* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python. +* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value. +* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly + access Java transforms defined in the allowlist from the Python side. + +Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below. + +##### Step 1 + +To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names, +constructor methods, and builder methods that are directly available to be used from the Python side. + +Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which +denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API. + +{{< highlight >}} +version: v1 +allowedClasses: +- className: my.beam.transforms.JavaDataGenerator + allowedConstructorMethods: + - create + allowedBuilderMethods: + - withJavaDataGeneratorConfig +{{< /highlight >}} + +##### Step 2 + +The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started +as a local Java process using the following command. + +{{< highlight >}} +java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file> +{{< /highlight >}} + +Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies Review comment: Done. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav #### 13.1.1. Creating cross-language Java transforms +There are two ways to make Java transforms available to other SDKs. + +* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code. +* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes. + +##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code + +Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example, +* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline +* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code + +> **Note:** This feature is currently only available when using Java transforms from a Python pipeline. + +To be eligible for direct usage, API of the Java transforms has to follow the following pattern. +* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class. +* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform. + +See below for the structure of an example Java class that can be directly used from the Python API. + +{{< highlight >}} +public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> { + . . . + + // Following method satisfies the Requirement 1. + // Note that you may also use a class constructor instead of a static method. + public static JavaDataGenerator create(Integer size) { + return new JavaDataGenerator(size); + } + + static class JavaDataGeneratorConfig implements Serializable { + public String prefix; + public long length; + public String suffix; + . . . + } + + // Following method conforms to the Requirement 2 + public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) { + return new JavaDataGenerator(this.size, javaDataGeneratorConfig); + } + + . . . +} +{{< /highlight >}} + +To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following. + +* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python. +* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value. +* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly + access Java transforms defined in the allowlist from the Python side. + +Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below. + +##### Step 1 + +To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names, +constructor methods, and builder methods that are directly available to be used from the Python side. + +Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which +denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API. + +{{< highlight >}} +version: v1 +allowedClasses: +- className: my.beam.transforms.JavaDataGenerator + allowedConstructorMethods: + - create + allowedBuilderMethods: + - withJavaDataGeneratorConfig +{{< /highlight >}} + +##### Step 2 + +The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started +as a local Java process using the following command. + +{{< highlight >}} +java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file> +{{< /highlight >}} + +Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies +if an expansion service address was not provided. + +##### Step 3 + +You can directly use the Java class from your Python pipeline using a stub transform created using the `JavaExternalTransform` API. This API allows you to construct the transform +using the Java class name and allows you to invoke builder methods to configure the class. + +Constructor and method parameter types are mapped between Python and Java using a Beam Schema. The Schema is auto-generated using the object types +provided in the Python side. If the Java class constructor method or builder method accepts any complex object types, make sure that Beam Schema Review comment: Done. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav #### 13.1.1. Creating cross-language Java transforms +There are two ways to make Java transforms available to other SDKs. + +* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code. +* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes. + +##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code + +Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example, +* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline +* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code + +> **Note:** This feature is currently only available when using Java transforms from a Python pipeline. + +To be eligible for direct usage, API of the Java transforms has to follow the following pattern. +* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class. +* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform. Review comment: Done. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav #### 13.1.1. Creating cross-language Java transforms +There are two ways to make Java transforms available to other SDKs. + +* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code. +* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes. Review comment: Done. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6752,6 +6861,43 @@ Currently Python external transforms are limited to dependencies available in co Go currently does not support creating cross-language transforms, only using cross-language transforms from other languages; see more at [BEAM-9923](https://issues.apache.org/jira/browse/BEAM-9923). +#### 13.1.4. Selecting a URN for Cross-language Transforms + +Developing a cross-language transform involves defining a URN for registering the transform with an expansion service. In this section +we provide a convention for defining such URNs. Following this convention is optional but following it will make sure that your transform +will not run into conflicts when registering in an expansion service along with transforms developed by other developers. + +##### Schema + +Suggested URN consists of the components given below. +* ns-id: A namespace identifier. Default recommendation is `beam:transform`. +* org-identifier: Identifies the organization where the transform was defined. Transforms defined in Apache Beam use `org.apache.beam` for this. +* functionality-identifier - Identifies the functionality of the cross-language transform. +* version - a version number for the transform + +We provide the schema from the URN convention in [augmented Backus–Naur](https://en.wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_form) form. +Keywords in upper case are from the [URN spec](https://datatracker.ietf.org/doc/html/rfc8141). + +{{< highlight >}} +transform-urn = ns-id “:” org-identifier “:” functionality-identifier “:” version +ns-id = (“beam” / NID) “:” “transform” +id-char = ALPHA / DIGIT / "-" / "." / "_" / "~" ; A subset of characters allowed in a URN +org-identifier = 1*id-char +functionality-identifier = 1*id-char +version = “v” 1*(DIGIT / “.”) ; For example, ‘v1.2’ +{{< /highlight >}} + +##### Examples + +Below we’ve given some example transforms classes and corresponding URNs to be used. Review comment: Done. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6752,6 +6861,43 @@ Currently Python external transforms are limited to dependencies available in co Go currently does not support creating cross-language transforms, only using cross-language transforms from other languages; see more at [BEAM-9923](https://issues.apache.org/jira/browse/BEAM-9923). +#### 13.1.4. Selecting a URN for Cross-language Transforms + +Developing a cross-language transform involves defining a URN for registering the transform with an expansion service. In this section +we provide a convention for defining such URNs. Following this convention is optional but following it will make sure that your transform +will not run into conflicts when registering in an expansion service along with transforms developed by other developers. + +##### Schema + +Suggested URN consists of the components given below. Review comment: Done. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav #### 13.1.1. Creating cross-language Java transforms +There are two ways to make Java transforms available to other SDKs. + +* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code. +* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes. + +##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code + +Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example, +* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline +* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code + +> **Note:** This feature is currently only available when using Java transforms from a Python pipeline. + +To be eligible for direct usage, API of the Java transforms has to follow the following pattern. +* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class. +* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform. + +See below for the structure of an example Java class that can be directly used from the Python API. + +{{< highlight >}} +public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> { + . . . + + // Following method satisfies the Requirement 1. + // Note that you may also use a class constructor instead of a static method. + public static JavaDataGenerator create(Integer size) { + return new JavaDataGenerator(size); + } + + static class JavaDataGeneratorConfig implements Serializable { + public String prefix; + public long length; + public String suffix; + . . . + } + + // Following method conforms to the Requirement 2 + public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) { + return new JavaDataGenerator(this.size, javaDataGeneratorConfig); + } + + . . . +} +{{< /highlight >}} + +To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following. + +* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python. +* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value. +* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly + access Java transforms defined in the allowlist from the Python side. + +Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below. + +##### Step 1 + +To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names, +constructor methods, and builder methods that are directly available to be used from the Python side. + +Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which +denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API. + +{{< highlight >}} +version: v1 +allowedClasses: +- className: my.beam.transforms.JavaDataGenerator + allowedConstructorMethods: + - create + allowedBuilderMethods: + - withJavaDataGeneratorConfig +{{< /highlight >}} + +##### Step 2 + +The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started +as a local Java process using the following command. + +{{< highlight >}} +java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file> +{{< /highlight >}} + +Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies +if an expansion service address was not provided. + +##### Step 3 + +You can directly use the Java class from your Python pipeline using a stub transform created using the `JavaExternalTransform` API. This API allows you to construct the transform +using the Java class name and allows you to invoke builder methods to configure the class. + +Constructor and method parameter types are mapped between Python and Java using a Beam Schema. The Schema is auto-generated using the object types +provided in the Python side. If the Java class constructor method or builder method accepts any complex object types, make sure that Beam Schema +for these objects are registered and available for the Java expansion service. If a schema has not been registered, Java expansion service will Review comment: Done. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav #### 13.1.1. Creating cross-language Java transforms +There are two ways to make Java transforms available to other SDKs. + +* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code. +* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes. + +##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code + +Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example, +* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline +* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code + +> **Note:** This feature is currently only available when using Java transforms from a Python pipeline. + +To be eligible for direct usage, API of the Java transforms has to follow the following pattern. +* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class. +* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform. + +See below for the structure of an example Java class that can be directly used from the Python API. + +{{< highlight >}} +public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> { + . . . + + // Following method satisfies the Requirement 1. + // Note that you may also use a class constructor instead of a static method. + public static JavaDataGenerator create(Integer size) { + return new JavaDataGenerator(size); + } + + static class JavaDataGeneratorConfig implements Serializable { + public String prefix; + public long length; + public String suffix; + . . . + } + + // Following method conforms to the Requirement 2 + public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) { + return new JavaDataGenerator(this.size, javaDataGeneratorConfig); + } + + . . . +} +{{< /highlight >}} + +To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following. + +* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python. +* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value. +* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly + access Java transforms defined in the allowlist from the Python side. + +Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below. + +##### Step 1 + +To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names, +constructor methods, and builder methods that are directly available to be used from the Python side. + +Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which +denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API. Review comment: Done. ########## File path: website/www/site/content/en/documentation/programming-guide.md ########## @@ -6752,6 +6861,43 @@ Currently Python external transforms are limited to dependencies available in co Go currently does not support creating cross-language transforms, only using cross-language transforms from other languages; see more at [BEAM-9923](https://issues.apache.org/jira/browse/BEAM-9923). +#### 13.1.4. Selecting a URN for Cross-language Transforms + +Developing a cross-language transform involves defining a URN for registering the transform with an expansion service. In this section +we provide a convention for defining such URNs. Following this convention is optional but following it will make sure that your transform Review comment: Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
