Hi Galen, > Gordon, is there a trick to running the sample code in flink-statefun-playground against yet-unreleased code that I'm missing?
You'd have to locally build an image from the release branch, with a temporary image version tag. Then, in the flink-statefun-playground, change the image versions in the docker-compose files to use that locally built image. IIRC, that's what we have been doing in the past. Admittedly, it's pretty manual - I don't think the CI manages this workflow. Thanks, Gordon On Mon, Aug 14, 2023 at 10:42 AM Galen Warren <ga...@cvillewarrens.com> wrote: > I created a pull request for this: [FLINK-31619] Upgrade Stateful > Functions to Flink 1.16.1 by galenwarren · Pull Request #331 · > apache/flink-statefun (github.com) > <https://github.com/apache/flink-statefun/pull/331>. > > JIRA is here: [FLINK-31619] Upgrade Stateful Functions to Flink 1.16.1 - > ASF JIRA (apache.org) > <https://issues.apache.org/jira/browse/FLINK-31619?filter=-1>. > > Statefun references 1.16.2, despite the title -- that version has come out > since the issue was created. > > I figured out how to run all the playground tests locally, but it took a > bit of reworking of the playground setup with respect to Docker; > specifically, the Docker contexts used to build the example functions > needed to be broadened (i.e. moved up the tree) so that, if needed, local > artifacts/code can be accessed from within the containers at build time. > Then I made the Docker compose.yml configurable through environment > variables to allow for them to run in either the original manner -- i.e. > pulling artifacts from public repos -- or in a "local" mode, where > artifacts are pulled from local builds. > > This process is a cleaner if the playground is a subfolder of the > flink-statefun project rather than be its own repository > (flink-statefun-playground), because then all the relative paths between > the playground files and the build artifacts are fixed. So, I'd like to > propose to move the playground files, modified as described above, to > flink-statefun/playground and retire flink-statefun-playground. I can > submit separate PR s those changes if everyone is on board. > > Also, should I plan to do the same upgrade to handle Flink 1.17.x? It > should be easy to do, especially while the 1.16.x upgrade is fresh on my > mind. > > Thanks. > > > On Fri, Aug 11, 2023 at 6:40 PM Galen Warren <ga...@cvillewarrens.com> > wrote: > >> I'm done with the code to make Statefun compatible with Flink 1.16, and >> all the tests (including e2e succeed). The required changes were pretty >> minimal. >> >> I'm running into a bit of a chicken/egg problem executing the tests in >> flink-statefun-playground >> <https://github.com/apache/flink-statefun-playground>, though. That >> project seems to assume that all the various Statefun artifacts are built >> and deployed to the various public repositories already. I've looked into >> trying to redirect references to local artifacts; however, that's also >> tricky since all the building occurs in Docker containers. >> >> Gordon, is there a trick to running the sample code in >> flink-statefun-playground against yet-unreleased code that I'm missing? >> >> Thanks. >> >> On Sat, Jun 24, 2023 at 12:40 PM Galen Warren <ga...@cvillewarrens.com> >> wrote: >> >>> Great -- thanks! >>> >>> I'm going to be out of town for about a week but I'll take a look at >>> this when I'm back. >>> >>> On Tue, Jun 20, 2023 at 8:46 AM Martijn Visser <mvis...@confluent.io> >>> wrote: >>> >>>> Hi Galen, >>>> >>>> Yes, I'll be more than happy to help with Statefun releases. >>>> >>>> Best regards, >>>> >>>> Martijn >>>> >>>> On Tue, Jun 20, 2023 at 2:21 PM Galen Warren <ga...@cvillewarrens.com> >>>> wrote: >>>> >>>>> Thanks. >>>>> >>>>> Martijn, to answer your question, I'd need to do a small amount of >>>>> work to get a PR ready, but not much. Happy to do it if we're deciding to >>>>> restart Statefun releases -- are we? >>>>> >>>>> -- Galen >>>>> >>>>> On Sat, Jun 17, 2023 at 9:47 AM Tzu-Li (Gordon) Tai < >>>>> tzuli...@apache.org> wrote: >>>>> >>>>>> > Perhaps he could weigh in on whether the combination of automated >>>>>> tests plus those smoke tests should be sufficient for testing with new >>>>>> Flink versions >>>>>> >>>>>> What we usually did at the bare minimum for new StateFun releases was >>>>>> the following: >>>>>> >>>>>> 1. Build tests (including the smoke tests in the e2e module, >>>>>> which covers important tests like exactly-once verification) >>>>>> 2. Updating the flink-statefun-playground repo and manually >>>>>> running all language examples there. >>>>>> >>>>>> If upgrading Flink versions was the only change in the release, I'd >>>>>> probably say that this is sufficient. >>>>>> >>>>>> Best, >>>>>> Gordon >>>>>> >>>>>> On Thu, Jun 15, 2023 at 5:25 AM Martijn Visser < >>>>>> martijnvis...@apache.org> wrote: >>>>>> >>>>>>> Let me know if you have a PR for a Flink update :) >>>>>>> >>>>>>> On Thu, Jun 8, 2023 at 5:52 PM Galen Warren via user < >>>>>>> user@flink.apache.org> wrote: >>>>>>> >>>>>>>> Thanks Martijn. >>>>>>>> >>>>>>>> Personally, I'm already using a local fork of Statefun that is >>>>>>>> compatible with Flink 1.16.x, so I wouldn't have any need for a >>>>>>>> released >>>>>>>> version compatible with 1.15.x. I'd be happy to do the PRs to modify >>>>>>>> Statefun to work with new versions of Flink as they come along. >>>>>>>> >>>>>>>> As for testing, Statefun does have unit tests and Gordon also sent >>>>>>>> me instructions a while back for how to do some additional smoke tests >>>>>>>> which are pretty straightforward. Perhaps he could weigh in on whether >>>>>>>> the >>>>>>>> combination of automated tests plus those smoke tests should be >>>>>>>> sufficient >>>>>>>> for testing with new Flink versions (I believe the answer is yes). >>>>>>>> >>>>>>>> -- Galen >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jun 8, 2023 at 8:01 AM Martijn Visser < >>>>>>>> martijnvis...@apache.org> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> Apologies for the late reply. >>>>>>>>> >>>>>>>>> I'm willing to help out with merging requests in Statefun to keep >>>>>>>>> them >>>>>>>>> compatible with new Flink releases and create new releases. I do >>>>>>>>> think that >>>>>>>>> validation of the functionality of these releases depends a lot on >>>>>>>>> those >>>>>>>>> who do these compatibility updates, with PMC members helping out >>>>>>>>> with the >>>>>>>>> formal process. >>>>>>>>> >>>>>>>>> > Why can't the Apache Software Foundation allow community members >>>>>>>>> to bring >>>>>>>>> it up to date? >>>>>>>>> >>>>>>>>> There's nothing preventing anyone from reviewing any of the >>>>>>>>> current PRs or >>>>>>>>> opening new ones. However, none of them are approved [1], so >>>>>>>>> there's also >>>>>>>>> nothing to merge. >>>>>>>>> >>>>>>>>> > I believe that there are people and companies on this mailing >>>>>>>>> list >>>>>>>>> interested in supporting Apache Flink Stateful Functions. >>>>>>>>> >>>>>>>>> If so, then now is the time to show. >>>>>>>>> >>>>>>>>> Would there be a preference to create a release with Galen's merged >>>>>>>>> compatibility update to Flink 1.15.2, or do we want to skip that >>>>>>>>> and go >>>>>>>>> straight to a newer version? >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> >>>>>>>>> Martijn >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> >>>>>>>>> https://github.com/apache/flink-statefun/pulls?q=is%3Apr+is%3Aopen+review%3Aapproved >>>>>>>>> >>>>>>>>> On Tue, Jun 6, 2023 at 3:55 PM Marco Villalobos < >>>>>>>>> mvillalo...@kineteque.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> > Why can't the Apache Software Foundation allow community members >>>>>>>>> to bring >>>>>>>>> > it up to date? >>>>>>>>> > >>>>>>>>> > What's the process for that? >>>>>>>>> > >>>>>>>>> > I believe that there are people and companies on this mailing >>>>>>>>> list >>>>>>>>> > interested in supporting Apache Flink Stateful Functions. >>>>>>>>> > >>>>>>>>> > You already had two people on this thread express interest. >>>>>>>>> > >>>>>>>>> > At the very least, we could keep the library versions up to date. >>>>>>>>> > >>>>>>>>> > There are only a small list of new features that might be >>>>>>>>> worthwhile: >>>>>>>>> > >>>>>>>>> > 1. event time processing >>>>>>>>> > 2. state rest api >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > On Jun 6, 2023, at 3:06 AM, Chesnay Schepler <ches...@apache.org> >>>>>>>>> wrote: >>>>>>>>> > >>>>>>>>> > If you were to fork it *and want to redistribute it* then the >>>>>>>>> short >>>>>>>>> > version is that >>>>>>>>> > >>>>>>>>> > 1. you have to adhere to the Apache licensing requirements >>>>>>>>> > 2. you have to make it clear that your fork does not belong >>>>>>>>> to the >>>>>>>>> > Apache Flink project. (Trademarks and all that) >>>>>>>>> > >>>>>>>>> > Neither should be significant hurdles (there should also be >>>>>>>>> plenty of >>>>>>>>> > online resources regarding 1), and if you do this then you can >>>>>>>>> freely share >>>>>>>>> > your fork with others. >>>>>>>>> > >>>>>>>>> > I've also pinged Martijn to take a look at this thread. >>>>>>>>> > To my knowledge the project hasn't decided anything yet. >>>>>>>>> > >>>>>>>>> > On 27/05/2023 04:05, Galen Warren wrote: >>>>>>>>> > >>>>>>>>> > Ok, I get it. No interest. >>>>>>>>> > >>>>>>>>> > If this project is being abandoned, I guess I'll work with my >>>>>>>>> own fork. Is >>>>>>>>> > there anything I should consider here? Can I share it with other >>>>>>>>> people who >>>>>>>>> > use this project? >>>>>>>>> > >>>>>>>>> > On Tue, May 16, 2023 at 10:50 AM Galen Warren < >>>>>>>>> ga...@cvillewarrens.com> <ga...@cvillewarrens.com> >>>>>>>>> > wrote: >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Hi Martijn, since you opened this discussion thread, I'm curious >>>>>>>>> what your >>>>>>>>> > thoughts are in light of the responses? Thanks. >>>>>>>>> > >>>>>>>>> > On Wed, Apr 19, 2023 at 1:21 PM Galen Warren < >>>>>>>>> ga...@cvillewarrens.com> <ga...@cvillewarrens.com> >>>>>>>>> > wrote: >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > I use Apache Flink for stream processing, and StateFun as a >>>>>>>>> hand-off >>>>>>>>> > >>>>>>>>> > point for the rest of the application. >>>>>>>>> > It serves well as a bridge between a Flink Streaming job and >>>>>>>>> > micro-services. >>>>>>>>> > >>>>>>>>> > This is essentially how I use it as well, and I would also be >>>>>>>>> sad to see >>>>>>>>> > it sunsetted. It works well; I don't know that there is a lot of >>>>>>>>> new >>>>>>>>> > development required, but if there are no new Statefun releases, >>>>>>>>> then >>>>>>>>> > Statefun can only be used with older Flink versions. >>>>>>>>> > >>>>>>>>> > On Tue, Apr 18, 2023 at 10:04 PM Marco Villalobos < >>>>>>>>> mvillalo...@kineteque.com> wrote: >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > I am currently using Stateful Functions in my application. >>>>>>>>> > >>>>>>>>> > I use Apache Flink for stream processing, and StateFun as a >>>>>>>>> hand-off >>>>>>>>> > point for the rest of the application. >>>>>>>>> > It serves well as a bridge between a Flink Streaming job and >>>>>>>>> > micro-services. >>>>>>>>> > >>>>>>>>> > I would be disappointed if StateFun was sunsetted. Its a good >>>>>>>>> idea. >>>>>>>>> > >>>>>>>>> > If there is anything I can do to help, as a contributor perhaps, >>>>>>>>> please >>>>>>>>> > let me know. >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > On Apr 3, 2023, at 2:02 AM, Martijn Visser < >>>>>>>>> martijnvis...@apache.org> <martijnvis...@apache.org> >>>>>>>>> > >>>>>>>>> > wrote: >>>>>>>>> > >>>>>>>>> > Hi everyone, >>>>>>>>> > >>>>>>>>> > I want to open a discussion on the status of the Statefun >>>>>>>>> Project [1] >>>>>>>>> > >>>>>>>>> > in Apache Flink. As you might have noticed, there hasn't been >>>>>>>>> much >>>>>>>>> > development over the past months in the Statefun repository [2]. >>>>>>>>> There is >>>>>>>>> > currently a lack of active contributors and committers who are >>>>>>>>> able to help >>>>>>>>> > with the maintenance of the project. >>>>>>>>> > >>>>>>>>> > In order to improve the situation, we need to solve the lack of >>>>>>>>> > >>>>>>>>> > committers and the lack of contributors. >>>>>>>>> > >>>>>>>>> > On the lack of committers: >>>>>>>>> > >>>>>>>>> > 1. Ideally, there are some of the current Flink committers who >>>>>>>>> have >>>>>>>>> > >>>>>>>>> > the bandwidth and can help with reviewing PRs and merging them. >>>>>>>>> > >>>>>>>>> > 2. If that's not an option, it could be a consideration that >>>>>>>>> current >>>>>>>>> > >>>>>>>>> > committers only approve and review PRs, that are approved by >>>>>>>>> those who are >>>>>>>>> > willing to contribute to Statefun and if the CI passes >>>>>>>>> > >>>>>>>>> > On the lack of contributors: >>>>>>>>> > >>>>>>>>> > 3. Next to having this discussion on the Dev and User mailing >>>>>>>>> list, we >>>>>>>>> > >>>>>>>>> > can also create a blog with a call for new contributors on the >>>>>>>>> Flink >>>>>>>>> > project website, send out some tweets on the Flink / Statefun >>>>>>>>> twitter >>>>>>>>> > accounts, post messages on Slack etc. In that message, we would >>>>>>>>> inform how >>>>>>>>> > those that are interested in contributing can start and where >>>>>>>>> they could >>>>>>>>> > reach out for more information. >>>>>>>>> > >>>>>>>>> > There's also option 4. where a group of interested people would >>>>>>>>> split >>>>>>>>> > >>>>>>>>> > Statefun from the Flink project and make it a separate top level >>>>>>>>> project >>>>>>>>> > under the Apache Flink umbrella (similar as recently has >>>>>>>>> happened with >>>>>>>>> > Flink Table Store, which has become Apache Paimon). >>>>>>>>> > >>>>>>>>> > If we see no improvements in the coming period, we should >>>>>>>>> consider >>>>>>>>> > >>>>>>>>> > sunsetting Statefun and communicate that clearly to the users. >>>>>>>>> > >>>>>>>>> > I'm looking forward to your thoughts. >>>>>>>>> > >>>>>>>>> > Best regards, >>>>>>>>> > >>>>>>>>> > Martijn >>>>>>>>> > >>>>>>>>> > [1] >>>>>>>>> https://nightlies.apache.org/flink/flink-statefun-docs-master/ < >>>>>>>>> > >>>>>>>>> > https://nightlies.apache.org/flink/flink-statefun-docs-master/> >>>>>>>>> > >>>>>>>>> > [2] https://github.com/apache/flink-statefun < >>>>>>>>> > >>>>>>>>> > https://github.com/apache/flink-statefun> >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> >>>>>>>>
