[
https://issues.apache.org/jira/browse/FLINK-19176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17194243#comment-17194243
]
Galen Warren commented on FLINK-19176:
--------------------------------------
Thanks for the quick reply! A few comments/questions:
{quote}Unfortunately adding ScalaPB payload serializer to the project directly
(even with runtime scope) will be very hard to get a consensus for, since
maintaining different Scala version is problematic and we have experienced that
in Flink in many different instances.
{quote}
Obviously, I'll defer to your judgment as to what could achieve consensus here.
It does look to me that StateFun already has a Scala dependency; the additional
ScalaPB dependency that would be added here (in statefun-flink-core) would just
reference the existing Scala dependency defined in statefun-parent, i.e.:
{noformat}
<dependency>
<groupId>com.thesamet.scalapb</groupId>
<artifactId>scalapb-runtime_${scala.binary.version}</artifactId>
<version>${scalapb.version}</version>
<scope>provided</scope>
</dependency>{noformat}
So it wouldn't be adding a new Scala dependency, but perhaps even referencing
the existing Scala dependency is undesirable. I do have a working version of
this locally, I'd be happy to share the code with you, if you want to see
exactly what it might look like.
{quote}I think that the long term solution should be a Scala native remote SDK,
and I would be perfectly happy to help kicking off this effort if you are
interested :)
{quote}
This sounds interesting but I don't quite understand what this would entail,
would you mind elaborating a bit?
{quote}Meanwhile, if you would like to use ScalaPB for the embedded functions,
then we would need to support pluggable MessagePayloadSerializer, that would be
picked up in runtime from the class-path. If you are interesting on working on
that, I can try to outline the steps needed to do that.
{quote}
This sounds promising to me! Are you suggesting that, instead of having to
choose among the predefined serializers (protobuf, kryo, etc.), one could
alternatively provide the name of a MessagePayloadSerializer class – i.e.
com.mydomain.MySerializer – in the StateFun configuration, and then
MessageFactory would create an instance of that class when serializers are
needed? That sounds like a great option to me. I'd be happy to work on that and
would appreciate any direction you could provide.
Thanks.
> Support ScalaPB as a message payload serializer in Stateful Functions
> ---------------------------------------------------------------------
>
> Key: FLINK-19176
> URL: https://issues.apache.org/jira/browse/FLINK-19176
> Project: Flink
> Issue Type: Improvement
> Components: Stateful Functions
> Affects Versions: 2.0.0
> Reporter: Galen Warren
> Priority: Major
> Fix For: statefun-2.1.0
>
>
> Currently, Stateful Functions supports four options for serialization of
> message payloads:
> * Protobuf (based on code generated for Java)
> * Kryo
> * Multilanguage
> * Raw
> This proposal is to add a fifth option to this list, to support serialization
> of message payloads based on [ScalaPB |[https://scalapb.github.io/docs/]].
> This would allow Scala developers to use ScalaPB-generated classes as message
> types in Stateful Functions directly as message types, without having to
> convert to/from code generated for Java and/or raw byte[] messages.
> This would be a simple implementation, as there is already a
> [MessagePayloadSerializer
> |[https://github.com/apache/flink-statefun/blob/8ffe619a94917d676cf1054c8a0e60de544663db/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/message/MessagePayloadSerializer.java]]
> interface that is implemented for each of the existing serialization
> methods; supporting ScalaPB would require a new class implementing
> MessagePayloadSerializer in terms of ScalaPB-generated code, and a means to
> select it, by adding a new value to the [MessageFactoryType
> |[https://github.com/apache/flink-statefun/blob/8ffe619a94917d676cf1054c8a0e60de544663db/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/message/MessageFactoryType.java]]enumeration.
> Plus testing :)
> I've done this already locally, the implementation is very similar to the
> existing [MessagePayloadSerializerPb
> |[https://github.com/apache/flink-statefun/blob/8ffe619a94917d676cf1054c8a0e60de544663db/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/message/MessagePayloadSerializerPb.java]]implementation.
> It uses a reference to ScalaPB in "provided" scope.
> Would you be interested in a pull request to add this? Primary benefit is to
> make it easy to use Stateful Functions in a Scala environment. Thanks.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
