sunjincheng created BEAM-7120:
---------------------------------
Summary: Turn `WindowedValue<T>` into `T` for all the interfaces
of Data Plane
Key: BEAM-7120
URL: https://issues.apache.org/jira/browse/BEAM-7120
Project: Beam
Issue Type: Improvement
Components: runner-core, sdk-java-harness, sdk-py-harness
Reporter: sunjincheng
Assignee: sunjincheng
The main purpose of this JIRA is Turn `WindowedValue<T>` into `T` for all the
interfaces of Data Plane.
The interface definitions for sending and receiving data in the code as follows:
- org.apache.beam.runners.fnexecution.data.FnDataService
{code:java}
public interface FnDataService {
<T> InboundDataClient receive(LogicalEndpoint inputLocation,
Coder<WindowedValue<T>> coder, FnDataReceiver<WindowedValue<T>> listener);
<T> CloseableFnDataReceiver<WindowedValue<T>> send(
LogicalEndpoint outputLocation, Coder<WindowedValue<T>> coder);
}
{code}
- org.apache.beam.fn.harness.data.BeamFnDataClient
{code:java}
public interface BeamFnDataClient {
<T> InboundDataClient receive(ApiServiceDescriptor apiServiceDescriptor,
LogicalEndpoint inputLocation, Coder<WindowedValue<T>> coder,
FnDataReceiver<WindowedValue<T>> receiver);
<T> CloseableFnDataReceiver<WindowedValue<T>> send(BeamFnDataGrpcClient
Endpoints.ApiServiceDescriptor apiServiceDescriptor, LogicalEndpoint
outputLocation, Coder<WindowedValue<T>> coder);
}
{code}
Both `Coder<WindowedValue<T>>` and `FnDataReceiver<WindowedValue<T>>` use
`WindowedValue` as the data structure that both sides of Runner and SDK Harness
know each other. Control Plane/Data Plane/State Plane/Logging is a highly
abstraction, such as Control Plane and Logging, these are common requirements
for all multi-language platforms. For example, the Flink community is also
discussing how to support Python UDF, as well as how to deal with docker
environment. how to data transfer, how to state access, how to logging etc. If
Beam can further abstract these service interfaces, i.e., interface definitions
are compatible with multiple engines, and finally provided to other projects in
the form of class libraries, it definitely will help other platforms that want
to support multiple languages. So we can further abstract the interface
definition of FnDataService's BeamFnDataClient, take the FnDataService#receive
interface as an example, and turn `WindowedValue<T>` into `T` so that other
platforms can be extended arbitrarily, as follows:
{code:java}
<T> InboundDataClient receive(LogicalEndpoint inputLocation, Coder<T> coder,
FnDataReceiver<T>> listener);
{code}
For details, see the mail discussion list:
[https://lists.apache.org/[email protected]:lte=1M:%5BDISCUSS%5D%20Turn%20%60WindowedValue]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
