Hi devs, I’d like to briefly provide an update / overview on what has been going on in the Flink Stateful Functions project since it was moved to Apache Flink [1].
One other main purpose of this is to figure out the next steps for the project, such as release timing, and some open questions surrounding updating the website, releasing Stateful Functions Python artifacts and Docker images, etc. *### Major new features* Since the last Stateful Functions release (which back then was still managed by Ververica, released roughly ~4 months ago), the community has added a few noticeable new features: Non-JVM language polyglot functions - This was top priority on the roadmap, and was successfully added to Stateful Functions over the past months. Some major additions under this category include: - Initial support for stateful polyglot functions, based on a HTTP-based request-reply protocol. - A convenience Python SDK that implements the request-reply protocol (PR opened [2], still pending merge). - Code-free way to bind functions / Kafka ingresses / Kafka egresses using textual YAML-based modules. Other noticeable additions - - Synchronous test utility for users to unit test their stateful functions - Additional state primitives such as PersistedAppendingBuffer, PersistedTable - Stateful Function’s variant of the State Processor API, currently supporting users to create Stateful Function application savepoints using Java. *### Release readiness* There has also been some work in getting the project in shape and ready to be released, both legal-wise and tooling-wise: - Documents are migrated to be running on Apache infra [3] [4]. - LICENSES and NOTICES have been added for sources and artifacts with bundled dependencies - Release utility scripts to create release branches / create source release / stage artifacts mostly adopted from the `apache/flink` repo. - We’re still missing release process documentation specifically for Stateful Functions in the community wiki, but it would make sense to add this along the way during the first release. *### Next steps?* With the amount of changes and new features since the last release, I think we’re at a good position to think about releasing the first version managed by the Flink community soon. What do you think? There are also still some open questions that needs to be discussed, such as: - The previous website for the project before the project was move to Apache Flink, https://statefun.io, still exists and points to the old docs. Should that be shutdown, or do we want to simply ask to have that redirected to the new docs running on Apache infra? - Do we want to update the Flink website’s main page navigation, so that the Stateful Functions docs are navigable from the Flink website? In general, how should the website look like if we add Stateful Functions downloads / docs? - Flink Stateful Functions will have Python packages that would be nice to be installable using PyPI. We can consider deploying that under the `pyflink` account [5], perhaps under a new project, e.g. pypi.org/project/apache-flink-statefun? - Flink Stateful Functions will also have a base Docker image that would also be nice to be officially endorsed by the Flink community. Would it make sense to incorporate this under the `apache/flink-docker` repo [6]? Please let us know any thoughts you may have! Cheers, Gordon [1] https://github.com/apache/flink-statefun [2] https://github.com/apache/flink-statefun/pull/52 [3] https://ci.apache.org/projects/flink/flink-statefun-docs-master/ [4] https://ci.apache.org/projects/flink/flink-statefun-docs-master/api/java/ [5] https://pypi.org/user/pyflink/ [6] https://github.com/apache/flink-docker
