asafm commented on code in PR #21992:
URL: https://github.com/apache/pulsar/pull/21992#discussion_r1493771613
##########
pip/pip-331.md:
##########
@@ -0,0 +1,129 @@
+# PIP-331: WASM Function API
+
+# Background knowledge
+
+WASM(WebAssembly) bytecode is designed to be encoded in a size- and
load-time-efficient binary format. WASM aims to leverage the common hardware
features available on various platforms to execute in browsers at machine code
speed.
+
+WASI(WebAssembly System Interface) provide a portable interface for
applications that run within a constrained sandbox environment, which allows
WASM to run in non browser environments such as Linux. It's portable and secure.
+
+# Motivation
+
+The server and client sides of the Pulsar function use protobuf for
decoupling. In principle, the language supported by protobuf can be supported
by the pulsar function, now Pulsar provided the java, python and golang
function client, but there are still many languages that are not supported.
Review Comment:
"provided the java, python and golang function client." - Do you mean
function runtime?
##########
pip/pip-331.md:
##########
@@ -0,0 +1,129 @@
+# PIP-331: WASM Function API
+
+# Background knowledge
+
+WASM(WebAssembly) bytecode is designed to be encoded in a size- and
load-time-efficient binary format. WASM aims to leverage the common hardware
features available on various platforms to execute in browsers at machine code
speed.
+
+WASI(WebAssembly System Interface) provide a portable interface for
applications that run within a constrained sandbox environment, which allows
WASM to run in non browser environments such as Linux. It's portable and secure.
+
+# Motivation
+
+The server and client sides of the Pulsar function use protobuf for
decoupling. In principle, the language supported by protobuf can be supported
by the pulsar function, now Pulsar provided the java, python and golang
function client, but there are still many languages that are not supported.
+
+Before all language adaptations are completed (and it's almost entirely
certain to be impossible), users cannot write pulsar function in their familiar
languages.
+
+# Goals
+
+## In Scope
+
+Other languages, as long as their code can be compiled into WASM bytecode
(such as Rust/golang/C++), users can use these languages to write pulsar
function.
+
+## Out of Scope
+
+All existing abilities of the Java pulsar function client are not
reimplemented, the WASM Pulsar functions is under the Java Pulsar functions.
+
+Due to the strict requirements of WASM on parameter types and for simplicity
reasons, types other than `java.lang.Long` are not used as parameters or return
value.
+
+# High Level Design
+
+```mermaid
+flowchart LR;
+
+ subgraph develop
+ direction TB
+ SourceCode ==> |"CompileToWASM"| WasmFile ==> |"RenameFile"|
MoveToTheResourceDirectory ==> UnitTest
+ end
+
+ subgraph runtime
+ direction TB
+ PulsarFunctionJava ==> |"LoadFromResource"| TheWasmFile ==> |"Invoke"|
TheSourceCode
+ end
+
+ develop --> runtime
+```
+
+# Detailed Design
+
+## Design & Implementation Details
+
+1. add `WasmLoader` to load WASM file and provide the WASM function to java,
also provide the java function to WASM if we need.
+
+2. add `AbstractWasmFunction` and `AbstractWasmWindowFunction` as the core
interface of the WASM function api.
+
+```java
+public abstract class AbstractWasmFunction<X, T> extends WasmLoader implements
Function<X, T> {
+
+ private static final String PROCESS_METHOD_NAME = "process";
+
+ protected static final String INITIALIZE_METHOD_NAME = "initialize";
+
+ protected static final String CLOSE_METHOD_NAME = "close";
+
+ protected static final Map<Long, Argument<?>> ARGUMENTS = new
ConcurrentHashMap<>();
+
+ @Override
+ public T process(X input, Context context) {
+ return super.getWasmExtern(PROCESS_METHOD_NAME)
+ .map(process -> {
+ Long argumentId = callWASI(input, context, process);
+ return doProcess(input, context, argumentId);
+ })
+ .orElseThrow(() -> new PulsarWasmException(
+ PROCESS_METHOD_NAME + " function not found in " +
super.getWasmName()));
+ }
+
+ private Long callWASI(X input,
+ Context context,
+ Extern process) {
+ // call WASI function
+ final Long argumentId = getArgumentId(input, context);
+ ARGUMENTS.put(argumentId, new Argument<>(input, context));
+ // WASI cannot easily pass Java objects like JNI, here we pass Long
+ // then we can get the argument by Long
+ WasmFunctions.consumer(super.getStore(), process.func(),
WasmValType.I64)
+ .accept(argumentId);
+ ARGUMENTS.remove(argumentId);
+ return argumentId;
+ }
+
+ protected abstract T doProcess(X input, Context context, Long argumentId);
+
+ protected abstract Long getArgumentId(X input, Context context);
+
+ @Override
+ public void initialize(Context context) {
+ super.getWasmExtern(INITIALIZE_METHOD_NAME)
+ .ifPresent(initialize -> callWASI(null, context, initialize));
+ }
+
+ @Override
+ public void close() {
+ super.getWasmExtern(CLOSE_METHOD_NAME)
+ .ifPresent(close -> callWASI(null, null, close));
+ super.close();
+ }
+
+ protected static class Argument<X> {
+ protected X input;
+ protected Context context;
+
+ private Argument(X input, Context context) {
+ this.input = input;
+ this.context = context;
+ }
+ }
+}
+```
+
+More detailed code implementation and test can be found in
[here](https://github.com/apache/pulsar/pull/21975)
+
+# Security Considerations
+
+Maybe need to add folders with tenancy name in the resource directory to
prevent conflicts between WASM file names of different tenancies.
Review Comment:
What is the resource directory? Is it shared today by Pulsar functions?
##########
pip/pip-331.md:
##########
@@ -0,0 +1,129 @@
+# PIP-331: WASM Function API
+
+# Background knowledge
+
+WASM(WebAssembly) bytecode is designed to be encoded in a size- and
load-time-efficient binary format. WASM aims to leverage the common hardware
features available on various platforms to execute in browsers at machine code
speed.
+
+WASI(WebAssembly System Interface) provide a portable interface for
applications that run within a constrained sandbox environment, which allows
WASM to run in non browser environments such as Linux. It's portable and secure.
+
+# Motivation
+
+The server and client sides of the Pulsar function use protobuf for
decoupling. In principle, the language supported by protobuf can be supported
by the pulsar function, now Pulsar provided the java, python and golang
function client, but there are still many languages that are not supported.
+
+Before all language adaptations are completed (and it's almost entirely
certain to be impossible), users cannot write pulsar function in their familiar
languages.
+
+# Goals
+
+## In Scope
+
+Other languages, as long as their code can be compiled into WASM bytecode
(such as Rust/golang/C++), users can use these languages to write pulsar
function.
+
+## Out of Scope
+
+All existing abilities of the Java pulsar function client are not
reimplemented, the WASM Pulsar functions is under the Java Pulsar functions.
+
+Due to the strict requirements of WASM on parameter types and for simplicity
reasons, types other than `java.lang.Long` are not used as parameters or return
value.
+
+# High Level Design
+
+```mermaid
+flowchart LR;
+
+ subgraph develop
+ direction TB
+ SourceCode ==> |"CompileToWASM"| WasmFile ==> |"RenameFile"|
MoveToTheResourceDirectory ==> UnitTest
+ end
+
+ subgraph runtime
+ direction TB
+ PulsarFunctionJava ==> |"LoadFromResource"| TheWasmFile ==> |"Invoke"|
TheSourceCode
+ end
+
+ develop --> runtime
+```
+
+# Detailed Design
+
+## Design & Implementation Details
+
+1. add `WasmLoader` to load WASM file and provide the WASM function to java,
also provide the java function to WASM if we need.
Review Comment:
The assumption here is that you plan to use an existing WASM Java library
that can load and invoke a WASM function?
##########
pip/pip-331.md:
##########
@@ -0,0 +1,129 @@
+# PIP-331: WASM Function API
+
+# Background knowledge
+
+WASM(WebAssembly) bytecode is designed to be encoded in a size- and
load-time-efficient binary format. WASM aims to leverage the common hardware
features available on various platforms to execute in browsers at machine code
speed.
+
+WASI(WebAssembly System Interface) provide a portable interface for
applications that run within a constrained sandbox environment, which allows
WASM to run in non browser environments such as Linux. It's portable and secure.
+
+# Motivation
+
+The server and client sides of the Pulsar function use protobuf for
decoupling. In principle, the language supported by protobuf can be supported
by the pulsar function, now Pulsar provided the java, python and golang
function client, but there are still many languages that are not supported.
+
+Before all language adaptations are completed (and it's almost entirely
certain to be impossible), users cannot write pulsar function in their familiar
languages.
+
+# Goals
+
+## In Scope
+
+Other languages, as long as their code can be compiled into WASM bytecode
(such as Rust/golang/C++), users can use these languages to write pulsar
function.
+
+## Out of Scope
+
+All existing abilities of the Java pulsar function client are not
reimplemented, the WASM Pulsar functions is under the Java Pulsar functions.
+
+Due to the strict requirements of WASM on parameter types and for simplicity
reasons, types other than `java.lang.Long` are not used as parameters or return
value.
Review Comment:
Parameters of the function? I thought Pulsar functions accept a message as a
parameter, and that's it, no? Can you provide some context on the parameters of
a function?
##########
pip/pip-331.md:
##########
@@ -0,0 +1,129 @@
+# PIP-331: WASM Function API
+
+# Background knowledge
+
+WASM(WebAssembly) bytecode is designed to be encoded in a size- and
load-time-efficient binary format. WASM aims to leverage the common hardware
features available on various platforms to execute in browsers at machine code
speed.
+
+WASI(WebAssembly System Interface) provide a portable interface for
applications that run within a constrained sandbox environment, which allows
WASM to run in non browser environments such as Linux. It's portable and secure.
+
+# Motivation
+
+The server and client sides of the Pulsar function use protobuf for
decoupling. In principle, the language supported by protobuf can be supported
by the pulsar function, now Pulsar provided the java, python and golang
function client, but there are still many languages that are not supported.
+
+Before all language adaptations are completed (and it's almost entirely
certain to be impossible), users cannot write pulsar function in their familiar
languages.
+
+# Goals
+
+## In Scope
+
+Other languages, as long as their code can be compiled into WASM bytecode
(such as Rust/golang/C++), users can use these languages to write pulsar
function.
+
+## Out of Scope
+
+All existing abilities of the Java pulsar function client are not
reimplemented, the WASM Pulsar functions is under the Java Pulsar functions.
+
+Due to the strict requirements of WASM on parameter types and for simplicity
reasons, types other than `java.lang.Long` are not used as parameters or return
value.
+
+# High Level Design
Review Comment:
It would be very useful if you could explain how these diagram stages work
and provide context.
1. Do we provide a template Maven / Gradle project that contains the build
plugin, which compiles to a WASM file?
2. Why must we move to a resource directory to run the unit test? Does the
build do it for us? Do we intend to provide a Unit test framework for the users
to execute or author the tests?
3. Can you expand on how the "LoadFromResource" works? Do you intend to use
an open-source Java library that does that? Which library is it?
##########
pip/pip-331.md:
##########
@@ -0,0 +1,129 @@
+# PIP-331: WASM Function API
+
+# Background knowledge
+
+WASM(WebAssembly) bytecode is designed to be encoded in a size- and
load-time-efficient binary format. WASM aims to leverage the common hardware
features available on various platforms to execute in browsers at machine code
speed.
+
+WASI(WebAssembly System Interface) provide a portable interface for
applications that run within a constrained sandbox environment, which allows
WASM to run in non browser environments such as Linux. It's portable and secure.
+
+# Motivation
+
+The server and client sides of the Pulsar function use protobuf for
decoupling. In principle, the language supported by protobuf can be supported
by the pulsar function, now Pulsar provided the java, python and golang
function client, but there are still many languages that are not supported.
+
+Before all language adaptations are completed (and it's almost entirely
certain to be impossible), users cannot write pulsar function in their familiar
languages.
+
+# Goals
+
+## In Scope
+
+Other languages, as long as their code can be compiled into WASM bytecode
(such as Rust/golang/C++), users can use these languages to write pulsar
function.
+
+## Out of Scope
+
+All existing abilities of the Java pulsar function client are not
reimplemented, the WASM Pulsar functions is under the Java Pulsar functions.
+
+Due to the strict requirements of WASM on parameter types and for simplicity
reasons, types other than `java.lang.Long` are not used as parameters or return
value.
+
+# High Level Design
+
+```mermaid
+flowchart LR;
+
+ subgraph develop
+ direction TB
+ SourceCode ==> |"CompileToWASM"| WasmFile ==> |"RenameFile"|
MoveToTheResourceDirectory ==> UnitTest
+ end
+
+ subgraph runtime
+ direction TB
+ PulsarFunctionJava ==> |"LoadFromResource"| TheWasmFile ==> |"Invoke"|
TheSourceCode
+ end
+
+ develop --> runtime
+```
+
+# Detailed Design
+
+## Design & Implementation Details
+
+1. add `WasmLoader` to load WASM file and provide the WASM function to java,
also provide the java function to WASM if we need.
+
+2. add `AbstractWasmFunction` and `AbstractWasmWindowFunction` as the core
interface of the WASM function api.
+
+```java
+public abstract class AbstractWasmFunction<X, T> extends WasmLoader implements
Function<X, T> {
+
+ private static final String PROCESS_METHOD_NAME = "process";
+
+ protected static final String INITIALIZE_METHOD_NAME = "initialize";
+
+ protected static final String CLOSE_METHOD_NAME = "close";
+
+ protected static final Map<Long, Argument<?>> ARGUMENTS = new
ConcurrentHashMap<>();
+
+ @Override
+ public T process(X input, Context context) {
+ return super.getWasmExtern(PROCESS_METHOD_NAME)
+ .map(process -> {
+ Long argumentId = callWASI(input, context, process);
Review Comment:
What exactly happens here? Does it load a virtual machine first that can
execute WASM code?
##########
pip/pip-331.md:
##########
@@ -0,0 +1,129 @@
+# PIP-331: WASM Function API
+
+# Background knowledge
+
+WASM(WebAssembly) bytecode is designed to be encoded in a size- and
load-time-efficient binary format. WASM aims to leverage the common hardware
features available on various platforms to execute in browsers at machine code
speed.
+
+WASI(WebAssembly System Interface) provide a portable interface for
applications that run within a constrained sandbox environment, which allows
WASM to run in non browser environments such as Linux. It's portable and secure.
+
+# Motivation
+
+The server and client sides of the Pulsar function use protobuf for
decoupling. In principle, the language supported by protobuf can be supported
by the pulsar function, now Pulsar provided the java, python and golang
function client, but there are still many languages that are not supported.
+
+Before all language adaptations are completed (and it's almost entirely
certain to be impossible), users cannot write pulsar function in their familiar
languages.
+
+# Goals
+
+## In Scope
+
+Other languages, as long as their code can be compiled into WASM bytecode
(such as Rust/golang/C++), users can use these languages to write pulsar
function.
+
+## Out of Scope
+
+All existing abilities of the Java pulsar function client are not
reimplemented, the WASM Pulsar functions is under the Java Pulsar functions.
+
+Due to the strict requirements of WASM on parameter types and for simplicity
reasons, types other than `java.lang.Long` are not used as parameters or return
value.
+
+# High Level Design
+
+```mermaid
+flowchart LR;
+
+ subgraph develop
+ direction TB
+ SourceCode ==> |"CompileToWASM"| WasmFile ==> |"RenameFile"|
MoveToTheResourceDirectory ==> UnitTest
+ end
+
+ subgraph runtime
+ direction TB
+ PulsarFunctionJava ==> |"LoadFromResource"| TheWasmFile ==> |"Invoke"|
TheSourceCode
+ end
+
+ develop --> runtime
+```
+
+# Detailed Design
+
+## Design & Implementation Details
+
+1. add `WasmLoader` to load WASM file and provide the WASM function to java,
also provide the java function to WASM if we need.
+
+2. add `AbstractWasmFunction` and `AbstractWasmWindowFunction` as the core
interface of the WASM function api.
+
+```java
+public abstract class AbstractWasmFunction<X, T> extends WasmLoader implements
Function<X, T> {
+
+ private static final String PROCESS_METHOD_NAME = "process";
+
+ protected static final String INITIALIZE_METHOD_NAME = "initialize";
+
+ protected static final String CLOSE_METHOD_NAME = "close";
+
+ protected static final Map<Long, Argument<?>> ARGUMENTS = new
ConcurrentHashMap<>();
+
+ @Override
+ public T process(X input, Context context) {
+ return super.getWasmExtern(PROCESS_METHOD_NAME)
+ .map(process -> {
+ Long argumentId = callWASI(input, context, process);
+ return doProcess(input, context, argumentId);
+ })
+ .orElseThrow(() -> new PulsarWasmException(
+ PROCESS_METHOD_NAME + " function not found in " +
super.getWasmName()));
+ }
+
+ private Long callWASI(X input,
+ Context context,
+ Extern process) {
+ // call WASI function
+ final Long argumentId = getArgumentId(input, context);
+ ARGUMENTS.put(argumentId, new Argument<>(input, context));
Review Comment:
Can you please elaborate more on how this handover is done?
Can WASM functions have access to Java variables? How?
Can the variables be of any type? How do you map a Java class to a WASM
class?
##########
pip/pip-331.md:
##########
@@ -0,0 +1,129 @@
+# PIP-331: WASM Function API
+
+# Background knowledge
+
+WASM(WebAssembly) bytecode is designed to be encoded in a size- and
load-time-efficient binary format. WASM aims to leverage the common hardware
features available on various platforms to execute in browsers at machine code
speed.
+
+WASI(WebAssembly System Interface) provide a portable interface for
applications that run within a constrained sandbox environment, which allows
WASM to run in non browser environments such as Linux. It's portable and secure.
+
+# Motivation
+
+The server and client sides of the Pulsar function use protobuf for
decoupling. In principle, the language supported by protobuf can be supported
by the pulsar function, now Pulsar provided the java, python and golang
function client, but there are still many languages that are not supported.
+
+Before all language adaptations are completed (and it's almost entirely
certain to be impossible), users cannot write pulsar function in their familiar
languages.
+
+# Goals
+
+## In Scope
+
+Other languages, as long as their code can be compiled into WASM bytecode
(such as Rust/golang/C++), users can use these languages to write pulsar
function.
+
+## Out of Scope
+
+All existing abilities of the Java pulsar function client are not
reimplemented, the WASM Pulsar functions is under the Java Pulsar functions.
Review Comment:
Can you expand on the last part of the sentence? What does it mean "the WASM
Pulsar functions is under the Java Pulsar functions."?
##########
pip/pip-331.md:
##########
@@ -0,0 +1,129 @@
+# PIP-331: WASM Function API
+
+# Background knowledge
+
+WASM(WebAssembly) bytecode is designed to be encoded in a size- and
load-time-efficient binary format. WASM aims to leverage the common hardware
features available on various platforms to execute in browsers at machine code
speed.
+
+WASI(WebAssembly System Interface) provide a portable interface for
applications that run within a constrained sandbox environment, which allows
WASM to run in non browser environments such as Linux. It's portable and secure.
+
+# Motivation
+
+The server and client sides of the Pulsar function use protobuf for
decoupling. In principle, the language supported by protobuf can be supported
by the pulsar function, now Pulsar provided the java, python and golang
function client, but there are still many languages that are not supported.
+
+Before all language adaptations are completed (and it's almost entirely
certain to be impossible), users cannot write pulsar function in their familiar
languages.
+
+# Goals
+
+## In Scope
+
+Other languages, as long as their code can be compiled into WASM bytecode
(such as Rust/golang/C++), users can use these languages to write pulsar
function.
+
+## Out of Scope
+
+All existing abilities of the Java pulsar function client are not
reimplemented, the WASM Pulsar functions is under the Java Pulsar functions.
+
+Due to the strict requirements of WASM on parameter types and for simplicity
reasons, types other than `java.lang.Long` are not used as parameters or return
value.
+
+# High Level Design
+
+```mermaid
+flowchart LR;
+
+ subgraph develop
+ direction TB
+ SourceCode ==> |"CompileToWASM"| WasmFile ==> |"RenameFile"|
MoveToTheResourceDirectory ==> UnitTest
+ end
+
+ subgraph runtime
+ direction TB
+ PulsarFunctionJava ==> |"LoadFromResource"| TheWasmFile ==> |"Invoke"|
TheSourceCode
+ end
+
+ develop --> runtime
+```
+
+# Detailed Design
+
+## Design & Implementation Details
+
+1. add `WasmLoader` to load WASM file and provide the WASM function to java,
also provide the java function to WASM if we need.
+
+2. add `AbstractWasmFunction` and `AbstractWasmWindowFunction` as the core
interface of the WASM function api.
+
+```java
+public abstract class AbstractWasmFunction<X, T> extends WasmLoader implements
Function<X, T> {
Review Comment:
I try to avoid giving inheritence as the technique for interfaces. It
contains too much knowledge for the user.
Can you explain why is it not possible for the interface of the user to only
be an interface?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
