Re: [DISCUSS] Enable scala formatting check

[Seth Wiesman]

Happy to help get this merged.

Do we want to wait until the 1.15 branch is cut? The change is mostly
trivial (reformatting) but does make changes to the build system.

Seth

On Wed, Mar 9, 2022 at 9:45 AM Francesco Guardiani 
wrote:

> Hi all,
> I've been spending some time prototyping a scalafmt conf, which doesn't
> look too different from our java style and tries to keep the same
> properties from our scalastyle conf. Here is the PR:
> https://github.com/apache/flink/pull/19025
>
> In particular, this is the scalafmt config commit:
>
> https://github.com/apache/flink/pull/19025/commits/cb32893df4b554e4526324c43c86681cc9fe8169
> And this is the commit removing scalastyle:
>
> https://github.com/apache/flink/pull/19025/commits/9ffe7d52e3368c5c40f15e3dc48f6d81691a8dd0
>
> I need some committer to pair with to merge the big PR, any volunteers? :)
>
> After we merge it I will also update the contributor guide doc to remove
> scalastyle.
>
> FG
>
> On Tue, Mar 8, 2022 at 10:07 AM David Anderson 
> wrote:
>
> > +1
> >
> > For flink-training we initially tried cloning the scalastyle setup from
> > flink, but we decided to use spotless + scalafmt instead.
> >
> > David
> >
> > On Mon, Mar 7, 2022 at 1:12 PM Timo Walther  wrote:
> >
> > > Big +1
> > >
> > > This will improve the contribution experience. Even though we stopped
> > > adding more Scala code, it is still necessary from time to time.
> > >
> > > Regards,
> > > Timo
> > >
> > > Am 02.03.22 um 09:29 schrieb 刘首维:
> > > > +1
> > > >
> > > >
> > > > I still remember my first pr. Lack of experience, I had to pay
> > attention
> > > to Scala code format and corrected the format manually, which made me a
> > > littleembarrassed(though I'm a big fan of Scala). I think this
> > > proposal will lighten the burden of writing Scala code.
> > > >
> > > >
> > > > Shouwei Liu
> > > >
> > > >
> > > > --原始邮件--
> > > > 发件人:
> > > "dev"
> > >   <
> > > kna...@apache.org;
> > > > 发送时间:2022年3月2日(星期三) 下午3:01
> > > > 收件人:"dev" > > >
> > > > 主题:Re: [DISCUSS] Enable scala formatting check
> > > >
> > > >
> > > >
> > > > +1 I've never written any Scala in Flink, but this makes a lot of
> sense
> > > to
> > > > me. Converging on a smaller set of tools and simplifying the build is
> > > > always a good idea and the Community already concluded before that
> > > spotless
> > > > is generally a good approach.
> > > >
> > > > On Tue, Mar 1, 2022 at 5:52 PM Francesco Guardiani <
> > > france...@ververica.com
> > > > wrote:
> > > >
> > > >  Hi all,
> > > > 
> > > >  I want to propose to enable the spotless scalafmt integration
> and
> > > remove
> > > >  the scalastyle plugin.
> > > > 
> > > >  From an initial analysis, scalafmt can do everything scalastyle
> > can
> > > do, and
> > > >  the integration with spotless looks easy to enable:
> > > > 
> https://github.com/diffplug/spotless/tree/main/plugin-maven#scala
> > .
> > > The
> > > >  scalafmt conf file gets picked up automatically from every IDE,
> > and
> > > it can
> > > >  be heavily tuned.
> > > > 
> > > >  This way we can unify the formatting and integrate with our CI
> > > without any
> > > >  additional configurations. And we won't need scalastyle anymore,
> > as
> > > >  scalafmt will take care of the checks:
> > > > 
> > > >  * mvn spotless:check will check both java and scala
> > > >  * mvn spotless:apply will format both java and scala
> > > > 
> > > >  WDYT?
> > > > 
> > > >  FG
> > > > 
> > > > 
> > > > 
> > > >  --
> > > > 
> > > >  Francesco Guardiani | Software Engineer
> > > > 
> > > >  france...@ververica.com
> > > > 
> > > > 
> > > >   > > > 
> > > >  Follow us @VervericaData
> > > > 
> > > >  --
> > > > 
> > > >  Join Flink Forward  > > Flink
> > > >  Conference
> > > > 
> > > >  Stream Processing | Event Driven | Real Time
> > > > 
> > > >  --
> > > > 
> > > >  Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> > > > 
> > > >  --
> > > > 
> > > >  Ververica GmbH
> > > > 
> > > >  Registered at Amtsgericht Charlottenburg: HRB 158244 B
> > > > 
> > > >  Managing Directors: Karl Anton Wehner, Holger Temme, Yip Park
> Tung
> > > Jason,
> > > >  Jinwei (Kevin) Zhang
> > > > 
> > > >
> > > >
> > >
> > >
> >
>

Re: [DISCUSS] CAST legacy behaviour

[Seth Wiesman]

+1

Especially as SQL upgrades are right around the corner, it makes sense to
get our defaults right.

Seth

On Mon, Feb 28, 2022 at 7:14 AM Martijn Visser 
wrote:

> +1 for setting this option to disabled by default. I believe failures
> should be brought forward as soon as possible, so they can be fixed as fast
> as possible. It will also be less confusing for new users. Last but not
> least, I believe the impact on existing users will be minimal (since it can
> be changed by changing one flag).
>
> Best regards,
>
> Martijn
>
> On Tue, 22 Feb 2022 at 17:55, Marios Trivyzas  wrote:
>
> > Thanks Francesco,
> >
> > The two arguments you posted, further strengthen the need to make it
> > DISABLED by default.
> >
> > On Tue, Feb 22, 2022 at 12:10 PM Francesco Guardiani <
> > france...@ververica.com> wrote:
> >
> > > Hi all,
> > > I'm +1 with what everything you said Marios.
> > >
> > > I'm gonna add another argument on top of that: the
> "legacy-cast-behavior"
> > > has also a broken type inference, leading to incorrect results or
> further
> > > errors down in the pipeline[1]. For example, take this:
> > >
> > > SELECT COALESCE(CAST('a' AS INT), 0) ...
> > >
> > > With the legacy cast behavior ENABLED, this is going to lead to the
> wrong
> > > result, as 'a' is inferred as VARCHAR NOT NULL, the CAST return value
> is
> > > inferred as INT NOT NULL, so the planner will drop COALESCE, and will
> > never
> > > return 0. Essentially, CAST will infer the wrong nullability leading to
> > > assigning its result to a NOT NULL type, when its value can effectively
> > be
> > > NULL.
> > >
> > > > You introduce a deprecated flag to help users
> > > using old versions of the software to smoothly transition to the new
> > > version, while the new users experience the new features/behavior,
> > > without the need to set a flag.
> > >
> > > This is IMO the major point on why we should disable the legacy cast
> > > behavior by default. This is even more relevant with 1.15 and the
> upgrade
> > > story, as per the problem described above, because users will now end
> up
> > > generating plans with wrong type inference, which will be hard to
> migrate
> > > in the next flink versions.
> > >
> > > FG
> > >
> > > [1] In case you're wondering why it wasn't fixed, the reason is that
> > fixing
> > > it means creating a breaking change, for details
> > > https://github.com/apache/flink/pull/18611#issuecomment-1028174877
> > >
> > >
> > > On Tue, Feb 22, 2022 at 10:07 AM Marios Trivyzas 
> > wrote:
> > >
> > > > Hello all!
> > > >
> > > > I would like to bring back the discussion regarding the
> > > > *table.exec.legacy-cast-behaviour*
> > > > configuration option which we are introducing with Flink *1.15*. This
> > > > option provides the users
> > > > with the flexibility to continue using the old (incorrect, according
> to
> > > SQL
> > > > standards) behaviour
> > > > of *CAST.*
> > > >
> > > > With Flink *1.15* we have introduced a bunch of fixes, improvements
> and
> > > new
> > > > casting functionality
> > > > between types, see https://issues.apache.org/jira/browse/FLINK-24403
> ,
> > > and
> > > > some of them are
> > > > guarded behind the legacy behaviour option:
> > > >
> > > >- Trimming and padding when casting to CHAR/VARCHAR types to
> respect
> > > the
> > > >specified length
> > > >- Changes for casting various types to CHAR/VARCHAR/STRING
> > > >- Runtime errors for CAST no longer emit *null *as result but
> > > exceptions
> > > >are thrown with a meaningful message for the cause, that fail the
> > > > pipeline. *TRY_CAST
> > > >*is introduced instead, which emits *null* results instead of
> > throwing
> > > >exceptions.
> > > >
> > > > Those changes become active if users set the
> > > > *table.exec.legacy-cast-behaviour
> > > > *option to *DISABLED*, otherwise
> > > > they will continue to experience the old, *erroneous*, behaviour of
> > > *CAST*.
> > > >
> > > > Currently, we have set the *table.exec.legacy-cast-behaviour *option
> > > > to be *ENABLED
> > > > *by default, so if users want
> > > > to get the new correct behaviour, they are required to set explicitly
> > the
> > > > option to *DISABLED*. Moreover, the option
> > > > itself is marked as deprecated, since the plan is to be removed in
> the
> > > > future, so that the old, erroneous behaviour
> > > > won't be an option, and the *TRY_CAST* would be the way to go if
> users
> > > > don't want to have errors and failed pipelines,
> > > > but have *null*s emitted in case of runtime errors when casting.
> > > >
> > > > I would like to start a discussion and maybe ask for voting, so that
> we
> > > set
> > > > the *table.exec.legacy-cast-behaviour* option
> > > > to *DISABLED *by default, so that users that want to keep their old
> > > > pipelines working the same way, without changing their
> > > > SQL/TableAPI code, would need to explicitly set it to *ENABLED.*
> > > >
> > > > My main argument for changing the default 

Re: [VOTE] Release 1.14.4, release candidate #1

[Seth Wiesman]

+1 non-binding

- built from source
- checked hashes and signatures
- started locally and deployed wordcount / stopped with savepoint /
restarted
- reviewed announcement post

Thanks for managing the release!

Seth

On Fri, Feb 25, 2022 at 7:30 AM Konstantin Knauf  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 1.14.4,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>  The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint 8C3FB007FE60 DEFA [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-1.14.4-rc1" [5],
> * website pull request listing the new release and adding announcement blog
> post [6].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Konstantin
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351231
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.14.4-rc1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1487/
> [5] https://github.com/apache/flink/tree/release-1.14.4-rc1
> [6] https://github.com/apache/flink-web/pull/510
>
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk
>

[jira] [Created] (FLINK-26374) JSON_OBJECT may throw NullPointerException on nullable column

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-26374:


 Summary: JSON_OBJECT may throw NullPointerException on nullable 
column 
 Key: FLINK-26374
 URL: https://issues.apache.org/jira/browse/FLINK-26374
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Runtime
Affects Versions: 1.15.0
Reporter: Seth Wiesman


>From ML: 

 

Using the latest SNAPSHOT BUILD.
 
If I have a column definition as
 
 .column(
                "events",
                DataTypes.ARRAY(
                    DataTypes.ROW(
                        DataTypes.FIELD("status", DataTypes.STRING().notNull()),
                        DataTypes.FIELD("timestamp", 
DataTypes.STRING().notNull()),
                        DataTypes.FIELD("increment_identifier", 
DataTypes.STRING().nullable()
 
And a query as
 
JSON_OBJECT('events' VALUE events) event_json
 
Will generate JSON correctly ONLY if increment_identifier is NOT NULL but will 
throw a NullPointerException on the first record that has that column as null.
 
Exception is not helpful.
 
Exception in thread "main" 
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
at 
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:141)
at 
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
at 
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at 
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at 
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:259)
at 
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at 
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at 
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at 
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
at 
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at 
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at 
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at 
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at 
org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
at akka.dispatch.OnComplete.internal(Future.scala:300)
at akka.dispatch.OnComplete.internal(Future.scala:297)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at 
org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
at 
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
at 
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
at 
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:24)
at 
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at 
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
at 
akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
at 
akka.dispat

Re: question about StatefunContext in golang Statefun SDK

[Seth Wiesman]

Hi all,

I believe the discussion revolved around:

1. fewer parameters
2. better aligned with other language sdks
3. we found precedent in other libraries (apologies this was long enough
ago I cannot remember which ones, I'm looking through old discussions now)

I would in general champion a solution that keeps the SDKs looking similar
across languages. A big part of statefun's positioning in the market is the
polyglot nature and making the transition between languages as seamless as
possible is very important.

Seth


On Tue, Feb 22, 2022 at 4:33 AM Till Rohrmann  wrote:

> Hi Galen,
>
> Thanks for explaining the problems with the current design. I think I've
> already learned quite a bit wrt Go thanks to you :-)
>
> From what you describe it seems indeed a bit restrictive to let the
> statefun.Context contain the context.Context w/o giving access to it. Maybe 
> @Seth
> Wiesman  can elaborate a bit more on the design
> decisions to make sure that we have the full picture.
>
> As a cheap workaround you could create a context.Context object by calling
> https://github.com/knative/pkg/blob/d48172451966/logging/logger.go#L45 on
> the statefun.Context and then pass this Context instance to the downstream
> function. But I agree that this is not the perfect solution.
>
> How do other libraries handle this situation if they offer a custom
> Context type? Maybe @Austin Cawley-Edwards  you
> have an opinion on the matter.
>
> Cheers,
> Till
>
> On Mon, Feb 21, 2022 at 7:42 PM Galen Warren 
> wrote:
>
>> So, upon further fiddling, I think it would be possible to keep full
>> backward compatibility and add the option for someone to use an Invoke
>> method with a separate context.Context and statefun.Runtime, via an
>> adapter, if direct manipulation of the context.Context is needed. So,
>> basically, the idea would be to let the user choose the form of the Invoke
>> method, with the default behavior remaining the same as now.
>>
>> This would require:
>>
>>- Recreating the Runtime interface (all methods currently defined on
>>Context except not embedding context.Context) and embedding it in the
>>statefun.Context interface, so that statefun.Context remains
>> effectively
>>unchanged
>>- Add StatefulFunctionV2 and StatefunFunctionV2Pointer to support the
>>new signature with separate context and runtime
>>- Add StatefulFunctionV2Adapter to wrap a StatefulFunctionV2 and expose
>>it as a StatefulFunction. The statefun.Context would get split into a
>>context.Context and a statefun.Runtime here in order to call the new
>>signature.
>>
>> Thoughts? I'd be happy to take a crack at it.
>>
>>
>> On Mon, Feb 21, 2022 at 12:06 PM Galen Warren 
>> wrote:
>>
>> > Was the reason to combine them the desire to have two parameters vs.
>> > three, or was there another motivation?
>> >
>> > On Mon, Feb 21, 2022 at 12:02 PM Seth Wiesman 
>> wrote:
>> >
>> >> FWIW I received a lot of early feedback explicitly asking me to couple
>> the
>> >> statefun specific operations with the Context (why the runtime type
>> went
>> >> away).
>> >>
>> >> Seth
>> >>
>> >> On Mon, Feb 21, 2022 at 10:32 AM Galen Warren > >
>> >> wrote:
>> >>
>> >> > Thanks for looking into this!
>> >> >
>> >> > The issue I think we'd run into with your proposal is that, often,
>> >> > libraries use non-exported types for context keys. Here is an example
>> >> > <
>> https://github.com/knative/pkg/blob/d48172451966/logging/logger.go#L45
>> >> >;
>> >> > in this case, the non-exported loggerKey{} is used as the key, inside
>> >> the
>> >> > exported WithLogger function. The key that would have to be supplied
>> to
>> >> the
>> >> > proposed Value and WithValue functions would not be accessible in
>> this
>> >> > case.
>> >> >
>> >> > Honestly, if *everything *were on the table -- and understand it very
>> >> well
>> >> > might not be -- I'd suggest decoupling the Golang context.Context and
>> >> the
>> >> > statefun Context, i.e. have two separate parameters to
>> >> > StatefulFunction.Invoke representing Golang context and statefun
>> >> > operations. This is actually how things were in an earlier version of
>> >> the
>> >> > Golang SDK; the first parameter to Invoke was the plain-vanill

Re: question about StatefunContext in golang Statefun SDK

[Seth Wiesman]

>> >
> > > >>> > You could then do a type assertion in the handlers for this type
> of
> > > >>> > context, and modify the context on it directly. It would be a bit
> > > >>> ugly, but
> > > >>> > may work.
> > > >>> >
> > > >>> > ```
> > > >>> > func (s aFunc) Invoke(ctx Context, message Message) error {
> > > >>> >   if sCtx, ok := ctx.(*statefun.StatefunContext); ok {
> > > >>> > sCtx.Context = context.WithValue(sCtx.Context, "logger",
> > aLogger)
> > > >>> >   }
> > > >>> >   // ...
> > > >>> > }
> > > >>> > ```
> > > >>> >
> > > >>> > Let me know what you all think,
> > > >>> > Austin
> > > >>> >
> > > >>> >
> > > >>> > [1]:
> > > >>> >
> > > >>>
> > >
> >
> https://github.com/apache/flink-statefun/blob/1dfe226d85fea05a46c8ffa688175b4c0f2d4900/statefun-sdk-go/v3/pkg/statefun/context.go#L66-L73
> > > >>> >
> > > >>> >
> > > >>> > On Fri, Feb 18, 2022 at 11:03 AM Galen Warren <
> > > ga...@cvillewarrens.com
> > > >>> >
> > > >>> > wrote:
> > > >>> >
> > > >>> >> Sorry Austin, I didn't see your response before I replied. Yes,
> > > we're
> > > >>> >> saying the same thing.
> > > >>> >>
> > > >>> >> On Fri, Feb 18, 2022 at 10:56 AM Austin Cawley-Edwards <
> > > >>> >> austin.caw...@gmail.com> wrote:
> > > >>> >>
> > > >>> >> > Hey all, jumping in. This makes sense to me – for instance to
> > > >>> attach a
> > > >>> >> > logger with some common metadata, e.g trace ID for the
> request?
> > > >>> This is
> > > >>> >> > common in go to add arbitrary items without updating the
> method
> > > >>> >> signatures,
> > > >>> >> > similar to thread local storage in Java.
> > > >>> >> >
> > > >>> >> > On Fri, Feb 18, 2022 at 10:53 AM Till Rohrmann <
> > > >>> trohrm...@apache.org>
> > > >>> >> > wrote:
> > > >>> >> >
> > > >>> >> > > Thanks for the clarification Galen. If you call the other Go
> > > >>> >> functions,
> > > >>> >> > > then you could also pass the other values as separate
> > arguments
> > > to
> > > >>> >> these
> > > >>> >> > > functions, can't you?
> > > >>> >> > >
> > > >>> >> > > Cheers,
> > > >>> >> > > Till
> > > >>> >> > >
> > > >>> >> > > On Fri, Feb 18, 2022 at 3:31 PM Galen Warren <
> > > >>> ga...@cvillewarrens.com
> > > >>> >> >
> > > >>> >> > > wrote:
> > > >>> >> > >
> > > >>> >> > > > The former.
> > > >>> >> > > >
> > > >>> >> > > > I think there's potential for confusion here because we're
> > > >>> using the
> > > >>> >> > > > word *function
> > > >>> >> > > > *in a couple of senses. One sense is a *stateful
> function*;
> > > >>> another
> > > >>> >> > sense
> > > >>> >> > > > is a *Go function*.
> > > >>> >> > > >
> > > >>> >> > > > What I'm looking to do is to put values in the Context so
> > that
> > > >>> >> > downstream
> > > >>> >> > > > Go functions that receive the context can access those
> > values.
> > > >>> Those
> > > >>> >> > > > downstream Go functions would be called during one
> > invocation
> > > >>> of the
> > > >>> >> > > > stateful function.
> > > >>> >> &

[jira] [Created] (FLINK-26200) Redirect statefun directly to docs

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-26200:


 Summary: Redirect statefun directly to docs
 Key: FLINK-26200
 URL: https://issues.apache.org/jira/browse/FLINK-26200
 Project: Flink
  Issue Type: Improvement
  Components: Project Website
Reporter: Seth Wiesman
Assignee: Seth Wiesman






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: question about StatefunContext in golang Statefun SDK

[Seth Wiesman]

Hi Galen,

No, that is not currently supported, the current idiomatic way would be to
pass those values to the struct implementing the Statefun interface.


type MyFunc struct { someRuntimeInfo string } func (m *MyFunc) Invoke(ctx
statefun.Context, message statefun.Message) error { } func main() { builder
:= statefun.StatefulFunctionsBuilder()
f := MyFunc { someRuntimeInfo: "runtime-provided" } builder.WithSpec
(statefun.StatefulFunctionSpec{ FunctionType: statefun.TypeNameFrom(
"example/my-func"), Function: f })
http.Handle("/statefun", builder.AsHandler())
_ = http.ListenAndServe(":8000", nil) }

Would this work for you? Or what is the context (pun intended) you are
looking for?

Seth

On Wed, Feb 16, 2022 at 4:35 PM Galen Warren 
wrote:

> When stateful functions are invoked, they are passed an instance of
> statefun.Context, which wraps the context.Context received by the HTTP
> request. Is there any way to customize that context.Context to, say, hold
> custom values, using ctx.WithValue()? I don't see a way but I wanted to
> ask.
>
> If not, would you be interested in a PR to add this functionality? A simple
> way might be to add a property to StatefulFunctionSpec, say:
>
> TransformContext func(ctx context.Context) context.Context
>
> ... that, if supplied, would be called to create a customized context that
> would be used downstream?
>
> Thanks.
>

Re: [VOTE] Remove Twitter connector

[Seth Wiesman]

+1 (binding)

On Thu, Feb 3, 2022 at 8:39 AM Fabian Paul  wrote:

> This connector is really a relict of the past.
>
> +1 (binding)
>
> Best,
> Fabian
>
> On Mon, Jan 31, 2022 at 11:47 AM Martijn Visser 
> wrote:
> >
> > Hi everyone,
> >
> > I would like to open up a vote to remove the Twitter connector in Flink
> > 1.15. This was brought up previously for a discussion [1].
> >
> > The vote will last for at least 72 hours, and will be accepted by
> > a consensus of active committers.
> >
> > Best regards,
> >
> > Martijn Visser
> > https://twitter.com/MartijnVisser82
> >
> > [1] https://lists.apache.org/thread/7sdvp4hj93rh0cz8r50800stzrpgkdm2
>

Re: [VOTE] Deprecate NiFi connector

[Seth Wiesman]

+1 (binding)

On Thu, Feb 3, 2022 at 8:44 AM Fabian Paul  wrote:

> Thanks for driving the deprecation efforts.
>
> +1 (binding)
>
> Best,
> Fabian
>
> On Mon, Jan 31, 2022 at 11:47 AM Martijn Visser 
> wrote:
> >
> > Hi everyone,
> >
> > I would like to open up a vote to deprecate NiFi in Flink 1.15 and remove
> > it in the next version. I've previously mentioned that we were looking
> for
> > maintainers for NiFi, but so far none have come forward [1].
> >
> > The vote will last for at least 72 hours, and will be accepted by a
> > consensus of active committers.
> >
> > Best regards,
> >
> > Martijn Visser
> > https://twitter.com/MartijnVisser82
> >
> > [1] https://lists.apache.org/thread/1vs3wmk66vsq6l4npjsfzltft4tz5tkq
>

Re: Need help with finding inner workings of watermark stream idleness

[Seth Wiesman]

Hi Jeff,

I think the class you're looking for is StatusWatermarkValve. Note that
this is fairly deep into the runtime stack.

Seth

On Tue, Feb 1, 2022 at 2:34 PM Jeff Carter  wrote:

> Thanks, Till.
>
> That definitely helps a bit. I'm still not seeing where there is some idle
> variable that the output.markIdle is setting to true (or whatever it sets).
> Like the ideal thing would be if there is just some "output.isIdle()" that
> could be called to know if the stream is or isnt idle. Since that doesn't
> exist, what is the variable in "output" that dictates if it is idle or not
> that that I'd just have to make an isIdle() method to make its state
> visible to other code.
>
> I see the checkIfIdle() method in the code (in at least the testing piece)
> you pointed out, but that seems like it's just a way to set a timer and
> check if the idle state should be set or not. But I dont know if that's
> setting some isIdle variable or if it's just checked and calculated
> everytime and that method is basically the variable I'm looking for. But
> that might just be my confusion.
>
> On Tue, Jan 11, 2022, 11:05 AM Till Rohrmann  wrote:
>
> > Hi Jeff,
> >
> > I think this happens in the WatermarksWithIdleness [1].
> >
> > [1]
> >
> >
> https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarksWithIdleness.java#L73
> >
> > Cheers,
> > Till
> >
> > On Tue, Jan 11, 2022 at 6:05 PM Jeff Carter 
> wrote:
> >
> > > I'm looking into making a feature for flink related to watermarks and
> am
> > > digging into the inner watermark mechanisms, specifically with
> idleness.
> > > I'm familiar with idleness, but digging into the root code I can only
> get
> > > to where idlenessTimeout gets set in
> WatermarkStrategyWithIdleness.java.
> > >
> > >  But what I'm looking for the pieces beyond that. If I set the idleness
> > to
> > > 500 milliseconds, where in the code does it actually go "I haven't
> seen a
> > > message in 500 milliseconds. I'm setting this stream to idle."?
> > >
> > > The reason being that what I'm thinking of would need to be able to see
> > if
> > > any streams are marked idle, and if so react accordingly.
> > >
> > > Thanks for any help in advance.
> > >
> >
>

Re: [VOTE] Apache Flink Stateful Functions 3.2.0, release candidate #1

[Seth Wiesman]

+1 (non-binding)

- verified checksums and signatures
- built from source and ran e2e tests
- checked dependencies / licenses
- deployed greeter with golang sdk
- reviewed blog post

Cheers,

Seth


On Fri, Jan 28, 2022 at 4:22 AM Igal Shilman  wrote:

> +1 (binding)
>
> - verified checksum and signatures
> - build from source code
> - run e2e tests
> - verified that there are no binary artifacts in the source release
> - built the python sdk from sources, and run the python greeter example
> - built from source the js sdk and run the javascript greeter example.
>
>
> Thanks,
> Igal.
>
>
>
>
>
>
> On Thu, Jan 27, 2022 at 8:20 PM Mingmin Xu  wrote:
>
> > +1 (non-binding)
> >
> > - verified checksum and signatures
> > - build from source code
> > - version checked
> > - test docker PR
> > - test flink-statefun-playground/greeter with 3.2.0
> >
> > Misc, do we want to upgrade flink-statefun-playground together? Currently
> > the README information is a little behind.
> >
> > B.R.
> > Mingmin
> >
> >
> > On Wed, Jan 26, 2022 at 4:55 AM Till Rohrmann 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > a quick update on the vote:
> > >
> > > The correct link for the artifacts at the Apache Nexus repository is
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1485/.
> > >
> > > Moreover, there is now also a tag for the GoLang SDK:
> > >
> https://github.com/apache/flink-statefun/tree/statefun-sdk-go/v3.2.0-rc1
> > .
> > >
> > > Cheers,
> > > Till
> > >
> > > On Tue, Jan 25, 2022 at 10:49 PM Till Rohrmann 
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release candidate #1 for the version
> > 3.2.0
> > > > of Apache Flink Stateful Functions, as follows:
> > > >
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > **Release Overview**
> > > >
> > > > As an overview, the release consists of the following:
> > > > a) Stateful Functions canonical source distribution, to be deployed
> to
> > > the
> > > > release repository at dist.apache.org
> > > > b) Stateful Functions Python SDK distributions to be deployed to PyPI
> > > > c) Maven artifacts to be deployed to the Maven Central Repository
> > > > d) New Dockerfiles for the release
> > > > e) GoLang SDK (contained in the repository)
> > > > f) JavaScript SDK (contained in the repository; will be uploaded to
> npm
> > > > after the release)
> > > >
> > > > **Staging Areas to Review**
> > > >
> > > > The staging areas containing the above mentioned artifacts are as
> > > follows,
> > > > for your review:
> > > > * All artifacts for a) and b) can be found in the corresponding dev
> > > > repository at dist.apache.org [2]
> > > > * All artifacts for c) can be found at the Apache Nexus Repository
> [3]
> > > >
> > > > All artifacts are signed with the key
> > > > B9499FA69EFF5DEEEBC3C1F5BA7E4187C6F73D82 [4]
> > > >
> > > > Other links for your review:
> > > > * JIRA release notes [5]
> > > > * source code tag "release-3.2.0-rc1" [6]
> > > > * PR for the new Dockerfiles [7]
> > > > * PR to update the website Downloads page to include Stateful
> Functions
> > > > links [8]
> > > > * GoLang SDK [9]
> > > > * JavaScript SDK [10]
> > > >
> > > > **Vote Duration**
> > > >
> > > > The voting time will run for at least 72 hours.
> > > > It is adopted by majority approval, with at least 3 PMC affirmative
> > > votes.
> > > >
> > > > Thanks,
> > > > Till
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release
> > > > [2]
> > > https://dist.apache.org/repos/dist/dev/flink/flink-statefun-3.2.0-rc1/
> > > > [3]
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1483/
> > > > [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [5]
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350540
> > > > [6] https://github.com/apache/flink-statefun/tree/release-3.2.0-rc1
> > > > [7] https://github.com/apache/flink-statefun-docker/pull/19
> > > > [8] https://github.com/apache/flink-web/pull/501
> > > > [9]
> > > >
> > >
> >
> https://github.com/apache/flink-statefun/tree/release-3.2.0-rc1/statefun-sdk-go
> > > > [10]
> > > >
> > >
> >
> https://github.com/apache/flink-statefun/tree/release-3.2.0-rc1/statefun-sdk-js
> > > >
> > >
> >
>

Re: [DISCUSS] Stateful Functions 3.2.0 release?

[Seth Wiesman]

+1

These are already a useful set of features to release to our users.

Seth

On Mon, Jan 24, 2022 at 8:45 AM Till Rohrmann  wrote:

> Hi everyone,
>
> We would like to kick off a new StateFun release 3.2.0. The new release
> will include the new JavaScript SDK and some useful major features:
>
> * JavaScript SDK [1]
> * Flink version upgrade to 1.14.3 [2]
> * Support different remote functions module names [3]
> * Allow creating custom metrics [4]
>
> The only missing ticket for this release is the documentation of the
> JavaScript SDK [5]. We plan to complete this in the next few days.
>
> Please let us know if you have any concerns.
>
> [1] https://issues.apache.org/jira/browse/FLINK-24256
> [2] https://issues.apache.org/jira/browse/FLINK-25708
> [3] https://issues.apache.org/jira/browse/FLINK-25308
> [4] https://issues.apache.org/jira/browse/FLINK-22533
> [5] https://issues.apache.org/jira/browse/FLINK-25775
>
> Cheers,
> Till
>

Re: [DISCUSS] FLIP-203: Incremental savepoints

[Seth Wiesman]

>> AFAIK state schema evolution should work both for native and canonical
>> savepoints.

Schema evolution does technically work for both formats, it happens after
the code paths have been unified, but the community has up until this point
considered that an unsupported feature. From my perspective making this
supported could be as simple as adding test coverage but that's an active
decision we'd need to make.

On Tue, Dec 21, 2021 at 7:43 AM Piotr Nowojski  wrote:

> Hi Konstantin,
>
> > In this context: will the native format support state schema evolution?
> If
> > not, I am not sure, we can let the format default to native.
>
> AFAIK state schema evolution should work both for native and canonical
> savepoints.
>
> Regarding what is/will be supported we will document as part of this
> FLIP-203. But it's not as simple as just the difference between native and
> canonical formats.
>
> Best, Piotrek
>
> pon., 20 gru 2021 o 14:28 Konstantin Knauf  napisał(a):
>
> > Hi Piotr,
> >
> > Thanks a lot for starting the discussion. Big +1.
> >
> > In my understanding, this FLIP introduces the snapshot format as a
> *really*
> > user facing concept. IMO it is important that we document
> >
> > a) that it is not longer the checkpoint/savepoint characteristics that
> > determines the kind of changes that a snapshots allows (user code, state
> > schema evolution, topology changes), but now this becomes a property of
> the
> > format regardless of whether this is a snapshots or a checkpoint
> > b) the exact changes that each format allows (code, state schema,
> topology,
> > state backend, max parallelism)
> >
> > In this context: will the native format support state schema evolution?
> If
> > not, I am not sure, we can let the format default to native.
> >
> > Thanks,
> >
> > Konstantin
> >
> >
> > On Mon, Dec 20, 2021 at 2:09 PM Piotr Nowojski 
> > wrote:
> >
> > > Hi devs,
> > >
> > > I would like to start a discussion about a previously announced follow
> up
> > > of the FLIP-193 [1], namely allowing savepoints to be in native format
> > and
> > > incremental. The changes do not seem invasive. The full proposal is
> > > written down as FLIP-203: Incremental savepoints [2]. Please take a
> look,
> > > and let me know what you think.
> > >
> > > Best,
> > > Piotrek
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership
> > > [2]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-203%3A+Incremental+savepoints#FLIP203:Incrementalsavepoints-Semantic
> > >
> >
> >
> > --
> >
> > Konstantin Knauf
> >
> > https://twitter.com/snntrable
> >
> > https://github.com/knaufk
> >
>

Re: [VOTE] Stateful functions 3.1.1 release

[Seth Wiesman]

+1 (non-binding)

- Verified signatures
- Checked diff
- Checked site PR
- Build from source and ran e2e tests

Seth

On Mon, Dec 20, 2021 at 10:59 AM Igal Shilman  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #2 for the version 3.1.1 of
> Apache Flink Stateful Functions, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> This release updates the Flink version to fix the log4j CVEs
>
> **Testing Guideline**
>
> You can find here [1] a page in the project wiki on instructions for
> testing.
> To cast a vote, it is not necessary to perform all listed checks,
> but please mention which checks you have performed when voting.
>
> **Release Overview**
>
> As an overview, the release consists of the following:
> a) Stateful Functions canonical source distribution, to be deployed to the
> release repository at dist.apache.org
> b) Stateful Functions Python SDK distributions to be deployed to PyPI
> c) Maven artifacts to be deployed to the Maven Central Repository
> d) New Dockerfiles for the release
> e) GoLang SDK tag v3.1.1-rc2
>
> **Staging Areas to Review**
>
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a) and b) can be found in the corresponding dev
> repository at dist.apache.org [2]
> * All artifacts for c) can be found at the Apache Nexus Repository [3]
>
> All artifacts are signed with the key
> 73BC0A2B04ABC80BF0513382B0ED0E338D622A92 [4]
>
> Other links for your review:
> * JIRA release notes [5]
> * source code tag "release-3.0.0-rc1" [6]
> * PR for the new Dockerfiles [7]
> * PR for the flink website [8]
>
> **Vote Duration**
>
> The voting time will run for 24 hours. We are targeting this vote to last
> until December. 21nd, 22:00 CET.
> Or It is adopted by majority approval, with at least 3 PMC affirmative
> votes.
>
> Thanks,
> Seth & Igal
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-3.1.1-rc2/
> [3] https://repository.apache.org/content/repositories/orgapacheflink-1466
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12351096==12315522
> [6] https://github.com/apache/flink-statefun/tree/release-3.1.1-rc2
> [7] https://github.com/apache/flink-statefun-docker/pull/18
> [8] https://github.com/apache/flink-web/pull/492
>

Re: [jira] [Created] (FLINK-25197) Using Statefun RequestReplyFunctionBuilder fails with Java 8 date/time type `java.time.Duration` not supported by default: add Module "org.apache.flink.shaded.jackso

[Seth Wiesman]

Hi Galen,

Sorry for the late reply, a lot of people are on sporadic schedules with
the holidays. A PR would be very welcome! I've gone ahead and assigned you
the Jira. And I'll watch the repo to review your pr.

cheers,

Seth

On Mon, Dec 20, 2021 at 8:24 AM Galen Warren 
wrote:

> I just wanted to check in on this, is there any interest in a PR to fix
> this issue? Looking at the master branch, it still looks to be a problem.
> Thanks.
>
> On Mon, Dec 6, 2021 at 9:52 AM Galen Warren (Jira) 
> wrote:
>
> > Galen Warren created FLINK-25197:
> > 
> >
> >  Summary: Using Statefun RequestReplyFunctionBuilder fails
> > with Java 8 date/time type `java.time.Duration` not supported by default:
> > add Module
> >
> "org.apache.flink.shaded.jackson2.com.fasterxml.jackson.datatype:jackson-datatype-jsr310"
> > to enable handling
> >  Key: FLINK-25197
> >  URL: https://issues.apache.org/jira/browse/FLINK-25197
> >  Project: Flink
> >   Issue Type: Bug
> >   Components: Stateful Functions
> > Affects Versions: statefun-3.1.0
> > Reporter: Galen Warren
> >  Fix For: statefun-3.1.0
> >
> >
> > When using RequestReplyFunctionBuilder to build a stateful functions job,
> > the job fails at runtime with:
> >
> > Java 8 date/time type `java.time.Duration` not supported by default: add
> > Module
> >
> "org.apache.flink.shaded.jackson2.com.fasterxml.jackson.datatype:jackson-datatype-jsr310"
> > to enable handling
> >
> > It appears this is because, in
> > [RequestReplyFunctionBuilder::transportClientPropertiesAsObjectNode|
> >
> https://github.com/apache/flink-statefun/blob/b4ba9547b8f0105a28544fd28a5e0433666e9023/statefun-flink/statefun-flink-datastream/src/main/java/org/apache/flink/statefun/flink/datastream/RequestReplyFunctionBuilder.java#L127
> ],
> > a default instance of ObjectMapper is used to serialize the client
> > properties, which now include a java.time.Duration. There is a
> > [StateFunObjectMapper|
> >
> https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-common/src/main/java/org/apache/flink/statefun/flink/common/json/StateFunObjectMapper.java
> ]
> > class in the project that has customized serde support, but it is not
> used
> > here.
> >
> > The fix seems to be to:
> >  * Use an instance of StateFunObjectMapper to serialize the client
> > properties in RequestReplyFunctionBuilder
> >  * Modify StateFunObjecdtMapper to both serialize and deserialize
> > instances of java.time.Duration (currently, only deserialization is
> > supported)
> >
> > I've made these changes locally and it seems to fix the problem. Would
> you
> > be interested in a PR? Thanks.
> >
> >
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.20.1#820001)
> >
>

Re: [DISCUSS] Immediate dedicated StateFun releases for log4j vulnerability

[Seth Wiesman]

And I'm happy to help with the release.

On Thu, Dec 16, 2021 at 12:55 PM Seth Wiesman  wrote:

> +1
>
> On Thu, Dec 16, 2021 at 12:37 PM Igal Shilman  wrote:
>
>> Hi All,
>>
>> Following the recent Apache Flink releases due to the log4j vulnerability,
>> I'd like to propose an immediate StateFun release - 3.1.1.
>> This release is basically the same as 3.1 but updates the Flink dependency
>> to 1.13.3.
>>
>> Please raise your concerns if any, otherwise we'll proceed with the
>> release.
>>
>> Thanks,
>> Igal.
>>
>

Re: [DISCUSS] Immediate dedicated StateFun releases for log4j vulnerability

[Seth Wiesman]

+1

On Thu, Dec 16, 2021 at 12:37 PM Igal Shilman  wrote:

> Hi All,
>
> Following the recent Apache Flink releases due to the log4j vulnerability,
> I'd like to propose an immediate StateFun release - 3.1.1.
> This release is basically the same as 3.1 but updates the Flink dependency
> to 1.13.3.
>
> Please raise your concerns if any, otherwise we'll proceed with the
> release.
>
> Thanks,
> Igal.
>

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

[Seth Wiesman]

+1 (non-binding)

- Checked diff of all versions and verified dep upgrade
- Verified checksum and signatures
- Built 1.14 from source
- checked blog post

Seth

On Wed, Dec 15, 2021 at 10:22 AM Yu Li  wrote:

> +1
>
> * Verified checksums and signatures
> * Reviewed website PR
>- Minor: left a comment to mention CVE-2021-45046
> * Checked and confirmed new tags only contain log4j version bump
> * Checked release notes and found no issues
>- I've moved FLINK-25317 to 1.14.3
>
> Thanks for driving these releases Chesnay!
>
> Best Regards,
> Yu
>
>
> On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler  wrote:
>
> > FYI; the publication of the python release for 1.11/1.12 will be delayed
> > because we hit the project size limit on pypi again, and increasing that
> > limit may take a while.
> > On the positive side, this gives us more time to fix the mac builds.
> >
> > On 15/12/2021 03:55, Chesnay Schepler wrote:
> > > Hi everyone,
> > >
> > > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and
> > > 1.14 to address CVE-2021-44228/CVE-2021-45046.
> > > It covers all 4 releases as they contain the same changes (upgrading
> > > Log4j to 2.16.0) and were prepared simultaneously by the same person.
> > > (Hence, if something is broken, it likely applies to all releases)
> > >
> > > Note: 1.11/1.12 are still missing the Python Mac releases.
> > >
> > >
> > > Please review and vote on the release candidate #1 for the versions
> > > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:
> > > [ ] +1, Approve the releases
> > > [ ] -1, Do not approve the releases (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > * JIRA release notes [1],
> > > * the official Apache source releases and binary convenience releases
> > > to be deployed to dist.apache.org [2], which are signed with the key
> > > with fingerprint C2EED7B111D464BA [3],
> > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > * source code tags [5],
> > > * website pull request listing the new releases and adding
> > > announcement blog post [6].
> > >
> > > The vote will be open for at least 24 hours. The minimum vote time has
> > > been shortened as the changes are minimal and the matter is urgent.
> > > It is adopted by majority approval, with at least 3 PMC affirmative
> > > votes.
> > >
> > > Thanks,
> > > Chesnay
> > >
> > > [1]
> > > 1.11:
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056
> > > 1.12:
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057
> > > 1.13:
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058
> > > 1.14:
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351059
> > > [2]
> > > 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/
> > > 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/
> > > 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/
> > > 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/
> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > [4]
> > > 1.11:
> > > https://repository.apache.org/content/repositories/orgapacheflink-1460
> > > 1.12:
> > > https://repository.apache.org/content/repositories/orgapacheflink-1462
> > > 1.13:
> > > https://repository.apache.org/content/repositories/orgapacheflink-1459
> > > 1.14:
> > > https://repository.apache.org/content/repositories/orgapacheflink-1461
> > > [5]
> > > 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1
> > > 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1
> > > 1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1
> > > 1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1
> > > [6] https://github.com/apache/flink-web/pull/489
> > >
> > >
> >
> >
>

[jira] [Created] (FLINK-25333) Harden UID management when constructing statefun universe

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-25333:


 Summary: Harden UID management when constructing statefun universe
 Key: FLINK-25333
 URL: https://issues.apache.org/jira/browse/FLINK-25333
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Seth Wiesman
Assignee: Seth Wiesman
 Fix For: statefun-3.2.0


As described in this stack overflow, we do not set UIDs on routers. Because of 
how the stream graph is generated, the uids may be non-deterministic. We should 
manually set all UIDs and enforce this via configuration. 

 

https://stackoverflow.com/questions/70316498/flink-statefun-high-availability-exception-java-lang-illegalstateexception-th



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [CANCELLED] Release 1.11.5/1.12.6/1.13.4/1.14.1, release candidate #1

[Seth Wiesman]

Thank you for managing these updates Chesnay!



On Tue, Dec 14, 2021 at 3:51 PM Chesnay Schepler  wrote:

> Since the maven artifacts have already been published we will use the
> next patch version for each release, i.e.:
> 1.11.6
> 1.12.7
> 1.13.5
> 1.14.2
>
> (We could technically just update the source/binaries, but that seems
> fishy).
>
> On 14/12/2021 22:38, Chesnay Schepler wrote:
> > I'm canceling the release because the issue was not fully fixed in
> > Log4j 2.15.0; see CVE-2021-45046.
> >
> > I will start preparing new release candidates that use Log4j 2.16.0 .
> >
> > On 14/12/2021 21:28, Chesnay Schepler wrote:
> >> The vote duration has passed and we have approved the releases.
> >>
> >> Binding votes:
> >> * Stephan
> >> * Till
> >> * Xintong
> >> * Zhu
> >> * Gordon
> >>
> >> I will not finalize the release.
> >>
> >> On 13/12/2021 20:28, Chesnay Schepler wrote:
> >>> Hi everyone,
> >>>
> >>> This vote is for the emergency patch releases for 1.11, 1.12, 1.13
> >>> and 1.14 to address CVE-2021-44228.
> >>> It covers all 4 releases as they contain the same changes (upgrading
> >>> Log4j to 2.15.0) and were prepared simultaneously by the same person.
> >>> (Hence, if something is broken, it likely applies to all releases)
> >>>
> >>> Please review and vote on the release candidate #1 for the versions
> >>> 1.11.5, 1.12.6, 1.13.4 and 1.14.1, as follows:
> >>> [ ] +1, Approve the releases
> >>> [ ] -1, Do not approve the releases (please provide specific comments)
> >>>
> >>> The complete staging area is available for your review, which includes:
> >>> * JIRA release notes [1],
> >>> * the official Apache source releases and binary convenience
> >>> releases to be deployed to dist.apache.org [2], which are signed
> >>> with the key with fingerprint C2EED7B111D464BA [3],
> >>> * all artifacts to be deployed to the Maven Central Repository [4],
> >>> * *the jars for 1.13/1.14 are still being built*
> >>> * source code tags [5],
> >>> * website pull request listing the new releases and adding
> >>> announcement blog post [6].
> >>>
> >>> The vote will be open for at least 24 hours. The minimum vote time
> >>> has been shortened as the changes are minimal and the matter is urgent.
> >>> It is adopted by majority approval, with at least 3 PMC affirmative
> >>> votes.
> >>>
> >>> Thanks,
> >>> Chesnay
> >>>
> >>> [1]
> >>> 1.11:
> >>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350327
> >>> 1.12:
> >>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350328
> >>> 1.13:
> >>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350686
> >>> 1.14:
> >>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350512
> >>> [2]
> >>> 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.5-rc1/
> >>> 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.6-rc1/
> >>> 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.4-rc1/
> >>> 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.1-rc1/
> >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >>> [4]
> >>> 1.11/1.12:
> >>> https://repository.apache.org/content/repositories/orgapacheflink-1455
> >>> 1.13:
> >>> https://repository.apache.org/content/repositories/orgapacheflink-1457
> >>> 1.14:
> >>> https://repository.apache.org/content/repositories/orgapacheflink-1456
> >>> [5]
> >>> 1.11: https://github.com/apache/flink/releases/tag/release-1.11.5-rc1
> >>> 1.12: https://github.com/apache/flink/releases/tag/release-1.12.6-rc1
> >>> 1.13: https://github.com/apache/flink/releases/tag/release-1.13.4-rc1
> >>> 1.14: https://github.com/apache/flink/releases/tag/release-1.14.1-rc1
> >>> [6] https://github.com/apache/flink-web/pull/489
> >>>
> >>
> >
>
>

Re: [VOTE] Release 1.11.5/1.12.6/1.13.4/1.14.1, release candidate #1

[Seth Wiesman]

+1 (non-binding)

- Checked Log4J version and updated license preambles on all releases
- Verified signatures on sources
- Reviewed blog post

Seth

On Mon, Dec 13, 2021 at 1:42 PM Jing Ge  wrote:

> +1   LGTM. Many thanks for your effort!
>
> On Mon, Dec 13, 2021 at 8:28 PM Chesnay Schepler 
> wrote:
>
> > Hi everyone,
> >
> > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and
> > 1.14 to address CVE-2021-44228.
> > It covers all 4 releases as they contain the same changes (upgrading
> > Log4j to 2.15.0) and were prepared simultaneously by the same person.
> > (Hence, if something is broken, it likely applies to all releases)
> >
> > Please review and vote on the release candidate #1 for the versions
> > 1.11.5, 1.12.6, 1.13.4 and 1.14.1, as follows:
> > [ ] +1, Approve the releases
> > [ ] -1, Do not approve the releases (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source releases and binary convenience releases to
> > be deployed to dist.apache.org [2], which are signed with the key with
> > fingerprint C2EED7B111D464BA [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> >  * *the jars for 1.13/1.14 are still being built*
> > * source code tags [5],
> > * website pull request listing the new releases and adding announcement
> > blog post [6].
> >
> > The vote will be open for at least 24 hours. The minimum vote time has
> > been shortened as the changes are minimal and the matter is urgent.
> > It is adopted by majority approval, with at least 3 PMC affirmative
> votes.
> >
> > Thanks,
> > Chesnay
> >
> > [1]
> > 1.11:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350327
> > 1.12:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350328
> > 1.13:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350686
> > 1.14:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350512
> > [2]
> > 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.5-rc1/
> > 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.6-rc1/
> > 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.4-rc1/
> > 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.1-rc1/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> > 1.11/1.12:
> > https://repository.apache.org/content/repositories/orgapacheflink-1455
> > 1.13:
> > https://repository.apache.org/content/repositories/orgapacheflink-1457
> > 1.14:
> > https://repository.apache.org/content/repositories/orgapacheflink-1456
> > [5]
> > 1.11: https://github.com/apache/flink/releases/tag/release-1.11.5-rc1
> > 1.12: https://github.com/apache/flink/releases/tag/release-1.12.6-rc1
> > 1.13: https://github.com/apache/flink/releases/tag/release-1.13.4-rc1
> > 1.14: https://github.com/apache/flink/releases/tag/release-1.14.1-rc1
> > [6] https://github.com/apache/flink-web/pull/489
> >
>

Re: [DISCUSS] Deprecate MapR FS

[Seth Wiesman]

+1

I actually thought we had already dropped this FS. If anyone is still
relying on it in production, the file system abstraction in Flink has been
incredibly stable over the years. They should be able to use the 1.14 MapR
FS with later versions of Flink.

Seth

On Wed, Dec 8, 2021 at 10:03 AM Martijn Visser 
wrote:

> Hi all,
>
> Flink supports multiple file systems [1] which includes MapR FS. MapR as a
> company doesn't exist anymore since 2019, the technology and intellectual
> property has been sold to Hewlett Packard.
>
> I don't think that there's anyone who's using MapR anymore and therefore I
> think it would be good to deprecate this for Flink 1.15 and then remove it
> in Flink 1.16. Removing this from Flink will slightly shrink the codebase
> and CI runtime.
>
> I'm also cross posting this to the User mailing list, in case there's
> still anyone who's using MapR.
>
> Best regards,
>
> Martijn
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>

[jira] [Created] (FLINK-24912) Migrate state processor API to DataStream API

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24912:


 Summary: Migrate state processor API to DataStream API
 Key: FLINK-24912
 URL: https://issues.apache.org/jira/browse/FLINK-24912
 Project: Flink
  Issue Type: Improvement
  Components: API / State Processor
Affects Versions: 1.15.0
Reporter: Seth Wiesman
Assignee: Jun Qin


Now that DataStream supports bounded execution, we should investigate migrating 
the State Processor API off DataSet. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24834) Add typed builders to DefaultRollingPolicy

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24834:


 Summary: Add typed builders to DefaultRollingPolicy
 Key: FLINK-24834
 URL: https://issues.apache.org/jira/browse/FLINK-24834
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / FileSystem
Affects Versions: 1.15.0
Reporter: Seth Wiesman
Assignee: Seth Wiesman


The default RollingPolicy is configured based on durations and memory sizes, 
but these parameters are specified via longs (milliseconds and bytes). We can 
provide a better user experience by adding methods that accept 
`java.time.Duration` and `org.apache.flink.configuration.MemorySize` instead. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24833) Prevent use of deprecated APIs in flink-examples

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24833:


 Summary: Prevent use of deprecated APIs in flink-examples
 Key: FLINK-24833
 URL: https://issues.apache.org/jira/browse/FLINK-24833
 Project: Flink
  Issue Type: Sub-task
  Components: Examples
Affects Versions: 1.15.0
Reporter: Seth Wiesman


We should be able to setup java compiler for examples
to fail on any usage of deprecated APIs.

Something along the lines of:


  maven-compiler-plugin
  ...
  
    
      compile
      process-sources
      
        compile
      
      
        -Xlint:deprecation
        true
      
    
  




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [NOTICE] Please keep flink-examples up to date

[Seth Wiesman]

@David Morávek  great idea! I've opened a subtask to add
that check after we fix the existing deprecations.

Seth

On Mon, Nov 8, 2021 at 12:55 PM David Morávek  wrote:

> Hi Seth, thanks for bringing this up. What do you think about placing a
> safeguard for this? We should be able to setup java compiler for examples
> to fail on any usage of deprecated APIs.
>
> Something along the lines of:
>
> 
>   maven-compiler-plugin
>   ...
>   
> 
>   compile
>   process-sources
>   
> compile
>   
>   
> -Xlint:deprecation
> true
>   
> 
>   
> 
>
> Best,
> D.
>
> On Mon, Nov 8, 2021 at 4:59 PM Till Rohrmann  wrote:
>
> > Thanks for raising awareness for keeping our examples up to date. I think
> > this is a very good practice and it also helps our users better
> understand
> > how to use Flink.
> >
> > Cheers,
> > Till
> >
> > On Mon, Nov 8, 2021 at 4:57 PM Fabian Paul 
> > wrote:
> >
> > > Hi Seth,
> > >
> > > Thanks for brining up this topic.
> > >
> > > Huge appreciations that you take this over initially and we should
> > > definitely
> > > take care as a community to what we show beginner users.
> > >
> > > We can also take the examples as show cases about things we have
> > developed
> > > and
> > > are proud of.
> > >
> > > Best,
> > > Fabian
> >
>

Re: [NOTICE] Please keep flink-examples up to date

[Seth Wiesman]

Also, please consider adding new examples if you are developing a new API!
It's a great addition to the documentation, and can help you workshop your
change from a usability perspective.

Seth

On Mon, Nov 8, 2021 at 9:34 AM Seth Wiesman  wrote:

> Hi Everyone,
>
> I noticed recently that many of the example applications in
> `flink-examples/` rely on deprecated apis. I am going to work on upgrading
> the examples for the current state of master. Going forward, if you
> deprecate a public API please try and double check if that API is being
> used in an example and update that example to the new version.
>
> Seth
>

[jira] [Created] (FLINK-24831) Upgrade DataStream Window examples

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24831:


 Summary: Upgrade DataStream Window examples
 Key: FLINK-24831
 URL: https://issues.apache.org/jira/browse/FLINK-24831
 Project: Flink
  Issue Type: Sub-task
  Components: Examples
Affects Versions: 1.15.0
Reporter: Seth Wiesman


Upgrade DataStream window examples to not rely on any deprecated APIs and work 
for both batch and streaming workloads. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24830) Update DataStream WordCount example

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24830:


 Summary: Update DataStream WordCount example
 Key: FLINK-24830
 URL: https://issues.apache.org/jira/browse/FLINK-24830
 Project: Flink
  Issue Type: Sub-task
  Components: Examples
Affects Versions: 1.15.0
Reporter: Seth Wiesman


Upgrade DataStream wordcount example to not use any deprecated API's and work 
for both batch and streaming workloads. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24829) Rename example directories based on API

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24829:


 Summary: Rename example directories based on API
 Key: FLINK-24829
 URL: https://issues.apache.org/jira/browse/FLINK-24829
 Project: Flink
  Issue Type: Sub-task
  Components: Examples
Affects Versions: 1.15.0
Reporter: Seth Wiesman


Rename example directories to better reflect the current state of the project; 
de-emphasis of dataset and stream / batch unification of datastream. 

 

flink-examples-batch --> flink-examples-dataset

flink-examples-streaming --> flink-examples-datastream

 

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[NOTICE] Please keep flink-examples up to date

[Seth Wiesman]

Hi Everyone,

I noticed recently that many of the example applications in
`flink-examples/` rely on deprecated apis. I am going to work on upgrading
the examples for the current state of master. Going forward, if you
deprecate a public API please try and double check if that API is being
used in an example and update that example to the new version.

Seth

[jira] [Created] (FLINK-24782) Create 1.14 release branch for training repo

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24782:


 Summary: Create 1.14 release branch for training repo
 Key: FLINK-24782
 URL: https://issues.apache.org/jira/browse/FLINK-24782
 Project: Flink
  Issue Type: Improvement
  Components: Documentation / Training / Exercises
Affects Versions: 1.14.0
Reporter: Seth Wiesman
Assignee: Seth Wiesman


Create release-1.14 branch for apache/flink-training



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24635) Clean up flink-examples

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24635:


 Summary: Clean up flink-examples
 Key: FLINK-24635
 URL: https://issues.apache.org/jira/browse/FLINK-24635
 Project: Flink
  Issue Type: Improvement
  Components: Examples
Reporter: Seth Wiesman
Assignee: Seth Wiesman
 Fix For: 1.15.0


The Flink DataStream examples have a number of deprecation warnings. These are 
some of the first things new users look at and we should be showing best 
practices. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24593) Create Table API Quickstart

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24593:


 Summary: Create Table API Quickstart
 Key: FLINK-24593
 URL: https://issues.apache.org/jira/browse/FLINK-24593
 Project: Flink
  Issue Type: Improvement
  Components: Quickstarts
Reporter: Seth Wiesman
Assignee: Seth Wiesman


Following FLINK-24516 we should add a Table API quickstart. The dependencies 
should be structured to easily run queries within the IDE but only package what 
should actually be included in the fat jar. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24516) Modernize Maven Archetype

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24516:


 Summary: Modernize Maven Archetype
 Key: FLINK-24516
 URL: https://issues.apache.org/jira/browse/FLINK-24516
 Project: Flink
  Issue Type: Improvement
  Components: Quickstarts
Reporter: Seth Wiesman


The maven archetypes used by many to start their first Flink application do not 
reflect the project's current state. 


Issues:
 * They still bundle the DataSet API and recommend it for batch processing
 * The JavaDoc recommends deprecated APIs

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24464) Metrics sometimes report negative backlog

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24464:


 Summary: Metrics sometimes report negative backlog
 Key: FLINK-24464
 URL: https://issues.apache.org/jira/browse/FLINK-24464
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Seth Wiesman


As report [here| 
[https://stackoverflow.com/questions/69441309/negative-backlog-in-apache-flink-stateful-functions]]
 the backlog metric sometimes reports a negative value. This is because the 
current count is stored in an in-memory variable, while the value used to 
decrement the count as the backlog clears is stored in Flink state. In the case 
of a job restart, the in-memory variable is reset to zero (it is a new 
instance) while the numbers in state are retained. 

 

A negative backlog is non-sensical, confusing, and should be prevented. 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24373) Remove BETA Tag from FLIP-27 Source Docs

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24373:


 Summary: Remove BETA Tag from FLIP-27 Source Docs
 Key: FLINK-24373
 URL: https://issues.apache.org/jira/browse/FLINK-24373
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Seth Wiesman
Assignee: Seth Wiesman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Drop Scala Shell

[Seth Wiesman]

+1

On Mon, Sep 20, 2021 at 6:04 AM Chesnay Schepler  wrote:

> +1
>
> On 20/09/2021 09:38, Martijn Visser wrote:
> > Hi all,
> >
> > I would like to start a vote on dropping the Scala Shell. This was
> > previously discussed in the mailing list [1]
> >
> > The vote will be open for at least 72 hours unless there is an objection
> or
> > not enough votes.
> >
> > Best regards,
> >
> > Martijn
> >
> > [1]
> >
> https://lists.apache.org/thread.html/rf7a7f935c43d3e98f94193be81b69f1c0d6e60b6fa09570531c3fa67%40%3Cdev.flink.apache.org%3E
> >
>
>

Re: [DISCUSS] Drop Scala Shell

[Seth Wiesman]

+1

The scala shell requires time and expertise that the contributors cannot
provide. It will be great to let the broader community have the opportunity
to foster and mature the scala shell outside the constraints imposed by the
ASF.

On Thu, Sep 16, 2021 at 9:02 AM Jeff Zhang  wrote:

> Hi Martijn,
>
> It is a pity that the flink community doesn't have resources to maintain
> scala shell which is useful for interactive user experience. We
> （Zeppelin）didn't use flink scala shell module directly, instead we
> customized the flink scala shell. So we are still interested in scala shell
> module, for us if no one want to maintain scala shell, it would be better
> to have it as flink 3rd party library, maybe in flink-packages.org
>
>
> Martijn Visser  于2021年9月16日周四 下午7:21写道：
>
> > Hi everyone,
> >
> > As was discussed quite a while ago, the community has planned to drop
> > support for Scala 2.11 [1]. I'm proposing to also drop the Scala Shell,
> > which was briefly discussed in the email thread for dropping Scala 2.11
> > [2].
> >
> > The Scala Shell doesn't work on Scala 2.12 [3] and there hasn't been much
> > traction to get this working and it looks like the Flink community /
> > committers don't want to maintain it anymore. Alternatively, if there's
> > still someone interested in the Scala Shell, it could also be moved to a
> > 3rd party repository. Removing the Scala Shell would/could help in
> getting
> > Scala 2.11 dropped quicker.
> >
> > Let me know your thoughts.
> >
> > Best regards,
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-20845
> > [2]
> >
> >
> https://lists.apache.org/thread.html/ra817c5b54e3de48d80e5b4e0ae67941d387ee25cf9779f5ae37d0486%40%3Cdev.flink.apache.org%3E
> > [3] https://issues.apache.org/jira/browse/FLINK-10911
> >
> >
> > Martijn Visser | Product Manager
> >
> > mart...@ververica.com
> >
> > 
> >
> >
> > Follow us @VervericaData
> >
> > --
> >
> > Join Flink Forward  - The Apache Flink
> > Conference
> >
> > Stream Processing | Event Driven | Real Time
> >
>
>
> --
> Best Regards
>
> Jeff Zhang
>

[jira] [Created] (FLINK-24303) SourceCoordinator exception may fail Session Cluster

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24303:


 Summary: SourceCoordinator exception may fail Session Cluster
 Key: FLINK-24303
 URL: https://issues.apache.org/jira/browse/FLINK-24303
 Project: Flink
  Issue Type: Bug
Reporter: Seth Wiesman


The SourceCoordinator currently forwards all exceptions from 
`Source#createEnumerator` up the stack triggering a JobMaster failover. 
However, JobMaster failover only works if HA is enabled[1]. If not, the fatal 
error handler will simply exit the JM process killing the entire cluster. 

 

It would be preferable if failure to create an enumerator did not take down the 
entire cluster, but instead failed that particular job. 

 

[1] 
https://github.com/apache/flink/blob/7f69331294ab2ab73f77b40a4320cdda53246afe/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L898-L903



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24301) use Async transport on all statefun playground examples

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24301:


 Summary: use Async transport on all statefun playground examples
 Key: FLINK-24301
 URL: https://issues.apache.org/jira/browse/FLINK-24301
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Seth Wiesman
Assignee: Seth Wiesman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24253) Load JdbcDialects via Service Loaders

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24253:


 Summary: Load JdbcDialects via Service Loaders
 Key: FLINK-24253
 URL: https://issues.apache.org/jira/browse/FLINK-24253
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / JDBC
Reporter: Seth Wiesman
Assignee: Seth Wiesman


The JDBC connector currently supports a hardcoded set of JDBC dialects. To 
support other JDBC datastores without adding additional feature complexity into 
Flink, we should allow plugging in additional dialects. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24110) Filter null ComponentJsonObjects

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24110:


 Summary: Filter null ComponentJsonObjects
 Key: FLINK-24110
 URL: https://issues.apache.org/jira/browse/FLINK-24110
 Project: Flink
  Issue Type: Improvement
Affects Versions: statefun-3.2.0
Reporter: Seth Wiesman
Assignee: Seth Wiesman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24109) Correct egress string serialization

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24109:


 Summary: Correct egress string serialization
 Key: FLINK-24109
 URL: https://issues.apache.org/jira/browse/FLINK-24109
 Project: Flink
  Issue Type: Improvement
Reporter: Seth Wiesman
Assignee: Seth Wiesman
 Fix For: statefun-3.2.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Apache Flink Stateful Functions 3.1.0, release candidate #1

[Seth Wiesman]

+1 (non-binding)

- verified signatures and hashes
- Checked licenses
- ran mvn clean install -Prun-e2e-tests
- ran golang greeter and showcase from the playground [1]

Seth

[1] https://github.com/apache/flink-statefun-playground/pull/12

On Wed, Aug 25, 2021 at 9:44 AM Igal Shilman  wrote:

> +1 from my side:
>
> Here are the results of my RC2 testing:
>
> - verified the signatures and hashes
> - verified that the source distribution doesn't contain any binary files
> - ran mvn clean install -Prun-e2e-tests
> - ran Java and Python greeters from the playground [1] with the new module
> structure, and async transport enabled.
> - verified that the docker image [2] builds and inspected the contents
> manually.
>
> Thanks,
> Igal
>
> [1] https://github.com/apache/flink-statefun-playground/tree/dev
> [2] https://github.com/apache/flink-statefun-docker/pull/15
>
>
> On Tue, Aug 24, 2021 at 3:34 PM Igal Shilman  wrote:
>
> > Sorry, the subject of the previous message should have said "[VOTE]
> Apache
> > Flink Stateful Functions 3.1.0, release candidate #2".
> >
> >
> > On Tue, Aug 24, 2021 at 3:24 PM Igal Shilman  wrote:
> >
> >> Hi everyone,
> >>
> >> Please review and vote on the release candidate #2 for the version 3.1.0
> >> of Apache Flink Stateful Functions, as follows:
> >> [ ] +1, Approve the release
> >> [ ] -1, Do not approve the release (please provide specific comments)
> >>
> >> **Testing Guideline**
> >>
> >> You can find here [1] a page in the project wiki on instructions for
> >> testing.
> >> To cast a vote, it is not necessary to perform all listed checks,
> >> but please mention which checks you have performed when voting.
> >>
> >> **Release Overview**
> >>
> >> As an overview, the release consists of the following:
> >> a) Stateful Functions canonical source distribution, to be deployed to
> >> the release repository at dist.apache.org
> >> b) Stateful Functions Python SDK distributions to be deployed to PyPI
> >> c) Maven artifacts to be deployed to the Maven Central Repository
> >> d) New Dockerfiles for the release
> >> e) GoLang SDK tag statefun-sdk-go/v3.1.0-rc2
> >>
> >> **Staging Areas to Review**
> >>
> >> The staging areas containing the above mentioned artifacts are as
> >> follows, for your review:
> >> * All artifacts for a) and b) can be found in the corresponding dev
> >> repository at dist.apache.org [2]
> >> * All artifacts for c) can be found at the Apache Nexus Repository [3]
> >>
> >> All artifacts are signed with the key
> >> 73BC0A2B04ABC80BF0513382B0ED0E338D622A92 [4]
> >>
> >> Other links for your review:
> >> * JIRA release notes [5]
> >> * source code tag "release-3.1.0-rc2" [6]
> >> * PR for the new Dockerfiles [7]
> >>
> >> **Vote Duration**
> >>
> >> The voting time will run for at least 72 hours (since RC1). We are
> >> targeting this vote to last until Thursday. 26th of August, 6pm CET.
> >> If it is adopted by majority approval, with at least 3 PMC affirmative
> >> votes, it will be released.
> >>
> >> Thanks,
> >> Igal
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release
> >> [2]
> >> https://dist.apache.org/repos/dist/dev/flink/flink-statefun-3.1.0-rc2/
> >> [3]
> >> https://repository.apache.org/content/repositories/orgapacheflink-1446/
> >> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> >> [5]
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12350038=12315522
> >> [6] https://github.com/apache/flink-statefun/tree/release-3.1.0-rc2
> >> [7] https://github.com/apache/flink-statefun-docker/pull/15
> >>
> >>
>

[jira] [Created] (FLINK-23976) Add additional availability timing metrics to Job lifecycle events

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-23976:


 Summary: Add additional availability timing metrics to Job 
lifecycle events
 Key: FLINK-23976
 URL: https://issues.apache.org/jira/browse/FLINK-23976
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Metrics
Reporter: Seth Wiesman


Flink currently contains a number of availability lifecycle metrics[1] showing 
how long it takes to move through different job status'. We propose adding two 
additional metrics; startingTime, and cancellingTime (open to bikeshedding on 
the metric names). 

 
 * startingTime is the time it takes a job to get to running. 
 * cancellingTime is the time spent in status CANCELLING 

 

 
[1]https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/metrics/#availability



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23951) Add golang examples to playground

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-23951:


 Summary: Add golang examples to playground
 Key: FLINK-23951
 URL: https://issues.apache.org/jira/browse/FLINK-23951
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Seth Wiesman
Assignee: Seth Wiesman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Apache Flink Stateful Functions 3.1.0, release candidate #1

[Seth Wiesman]

-1

There is an issue with the `go.mod` file which prevents it from properly
being released. I have opened a PR[1] .

This should not affect testing, this fix is only required to actually
publish the release.

[1] https://github.com/apache/flink-statefun/pull/260

On Mon, Aug 23, 2021 at 10:38 AM Igal Shilman  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 3.1.0 of
> Apache Flink Stateful Functions, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> **Testing Guideline**
>
> You can find here [1] a page in the project wiki on instructions for
> testing.
> To cast a vote, it is not necessary to perform all listed checks,
> but please mention which checks you have performed when voting.
>
> **Release Overview**
>
> As an overview, the release consists of the following:
> a) Stateful Functions canonical source distribution, to be deployed to the
> release repository at dist.apache.org
> b) Stateful Functions Python SDK distributions to be deployed to PyPI
> c) Maven artifacts to be deployed to the Maven Central Repository
> d) New Dockerfiles for the release
> e) GoLang SDK tag v3.1.0-rc1
>
> **Staging Areas to Review**
>
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a) and b) can be found in the corresponding dev
> repository at dist.apache.org [2]
> * All artifacts for c) can be found at the Apache Nexus Repository [3]
>
> All artifacts are signed with the key
> 73BC0A2B04ABC80BF0513382B0ED0E338D622A92 [4]
>
> Other links for your review:
> * JIRA release notes [5]
> * source code tag "release-3.1.0-rc1" [6]
> * PR for the new Dockerfiles [7]
>
> **Vote Duration**
>
> The voting time will run for at least 72 hours. We are targeting this vote
> to last until Thursday, 26th of August, 6pm CET.
> If it is adopted by majority approval, with at least 3 PMC affirmative
> votes, it will be released.
>
> Thanks,
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-3.1.0-rc1/
> [3]
> https://repository.apache.org/content/repositories/orgapacheflink-1445/
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12350038=12315522
> [6] https://github.com/apache/flink-statefun/tree/release-3.1.0-rc1
> [7] https://github.com/apache/flink-statefun-docker/pull/15
>

[jira] [Created] (FLINK-23927) Fix go.mod file for SDK release

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-23927:


 Summary: Fix go.mod file for SDK release
 Key: FLINK-23927
 URL: https://issues.apache.org/jira/browse/FLINK-23927
 Project: Flink
  Issue Type: Bug
  Components: Stateful Functions
Reporter: Seth Wiesman
Assignee: Seth Wiesman
 Fix For: statefun-3.1.0


To release the go SDK, the go.mod file needs to be in the `v3` subdirectory. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23878) Minor improvements to golang sdk

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-23878:


 Summary: Minor improvements to golang sdk
 Key: FLINK-23878
 URL: https://issues.apache.org/jira/browse/FLINK-23878
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Seth Wiesman
Assignee: Seth Wiesman
 Fix For: statefun-3.1.0


Misc enhancements to the golang SDK
 # Make cell.Write resumable
 # Improve serialization performance for common primitives



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Stateful Functions 3.1.0 release?

[Seth Wiesman]

+1 for the release



On Mon, Aug 16, 2021 at 8:25 AM Gordon Tai (戴資力)  wrote:

> Hi Igal,
>
> Thanks a lot for starting this discussion!
>
> The list of features sounds like this will be a great release. As the list
> is already quite packed with major features, +1 to feature freeze on August
> 20 so we can move towards 3.1.0.
>
> Thanks,
> Gordon
>
> On Mon, Aug 16, 2021, 18:57 Igal Shilman  wrote:
>
> > Hi everyone!
> >
> > We would like to kick off a new StateFun release (3.1.0), this release
> will
> > have some major features that users would benefit from, such as:
> >
> > * [FLINK-21308] Support delayed message cancellation.
> > * [FLINK-23296] Support pluggable transports for remote functions
> > invocation.
> > * [FLINK-23711] Add a non blocking, asynchronous transport for remote
> > function invocation.
> > * [FLINK-23600] Adopting kubernetes style resource formats for the
> > module.yaml definition.
> > * [FLINK-18810] Add a GoLang SDK for remote functions.
> >
> > We would like to propose a feature freeze for this Friday 20th of Aug,
> and
> > would like to kick off the release candidates early next week.
> >
> > Please let us know if you have any concerns.
> >
> > Thanks,
> > Igal
> >
>

[jira] [Created] (FLINK-23728) Support new state backend factory stack in State Processor API

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-23728:


 Summary: Support new state backend factory stack in State 
Processor API 
 Key: FLINK-23728
 URL: https://issues.apache.org/jira/browse/FLINK-23728
 Project: Flink
  Issue Type: Improvement
  Components: API / State Processor
Reporter: Seth Wiesman
Assignee: Seth Wiesman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23547) StateFun does not execute integration tests

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-23547:


 Summary: StateFun does not execute integration tests
 Key: FLINK-23547
 URL: https://issues.apache.org/jira/browse/FLINK-23547
 Project: Flink
  Issue Type: Bug
Reporter: Seth Wiesman
Assignee: Seth Wiesman


Stateful Functions is missing the surefire maven plugin and integration tests, 
marked with ITCase suffix, are not executed on install. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23520) Improve StateFun <-> DataStream Interop

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-23520:


 Summary: Improve StateFun <-> DataStream Interop
 Key: FLINK-23520
 URL: https://issues.apache.org/jira/browse/FLINK-23520
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Seth Wiesman
Assignee: Seth Wiesman


Improve Stateful Functions <-> DataStream interop to improve usable with remote 
functions. In particular: 

 
 * Automatic type conversion between JVM primitives and remote type system
 * Plugable types for use with complex class
 * Hide internal classes like TypedValue
 * Support all endpoint configuration including wildcards and URL path 
templating



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-182: Support watermark alignment of FLIP-27 Sources

[Seth Wiesman]

+1 (binding)

Seth

On Fri, Jul 23, 2021 at 7:31 AM Arvid Heise  wrote:

> +1 (binding)
>
> On Wed, Jul 21, 2021 at 1:55 PM Piotr Nowojski 
> wrote:
>
> > Hi everyone,
> >
> > I would like to start a vote on the FLIP-182 [1] which was discussed in
> > this
> > thread [2].
> > The vote will be open for at least 72 hours unless there is an objection
> > or not enough votes.
> >
> > Best,
> > Piotrek
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources
> > [2]
> >
> >
> https://mail-archives.apache.org/mod_mbox/flink-dev/202107.mbox/%3CCABpD1RUHJURJg7Vkbq4Tjz2yecd8Wv8kJiT46M11F-ODSakgcw%40mail.gmail.com%3E
> >
>

[jira] [Created] (FLINK-23439) Update "Local Installation" Page

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-23439:


 Summary: Update "Local Installation" Page
 Key: FLINK-23439
 URL: https://issues.apache.org/jira/browse/FLINK-23439
 Project: Flink
  Issue Type: Improvement
  Components: chinese-translation
Reporter: Seth Wiesman


Sync translation with ae136d18d84ed4d4ff631e990fd57e8c1867b30f



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Incrementally deprecating the DataSet API

[Seth Wiesman]

Happy to help clarify, and certainly, I don't think anyone would reject the
contribution if you wanted to add that functionality. The important point
we want to communicate is that the table API has matured to a level where
we do not see it as supplementary to DataStream but as an equal first-class
interface. Both optimize towards different making specific use cases easier
- declarative definition of pipelines vs fine-grained control - and neither
needs to support every operation natively because it should be seamless to
interoperate them.

Seth

On Fri, Jul 9, 2021 at 10:35 AM Etienne Chauchot 
wrote:

> Hi Seth,
>
> Thanks for your comment.
>
> That is perfect then !
>
> If the community agrees to give precedence to the table API connectors
> over the DataStream connectors as discussed here, all the features are
> already available then.
>
> I will not need to port my latest additions to table connectors then.
>
> Best
>
> Etienne
>
> On 08/07/2021 17:28, Seth Wiesman wrote:
> > Hi Etienne,
> >
> > The `toDataStream` method supports converting to concrete Java types, not
> > just Row, which can include your Avro specific-records. See example 2:
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/data_stream_api/#examples-for-todatastream
> >
> > On Thu, Jul 8, 2021 at 5:11 AM Etienne Chauchot 
> > wrote:
> >
> >> Hi Timo,
> >>
> >> Thanks for your answers, no problem with the delay, I was in vacation
> >> too last week :)
> >>
> >> My comments are inline
> >>
> >> Best,
> >>
> >> Etienne
> >>
> >> On 07/07/2021 16:48, Timo Walther wrote:
> >>> Hi Etienne,
> >>>
> >>> sorry for the late reply due to my vacation last week.
> >>>
> >>> Regarding: "support of aggregations in batch mode for DataStream API
> >>> [...] is there a plan to solve it before the actual drop of DataSet
> API"
> >>>
> >>> Just to clarify it again: we will not drop the DataSet API any time
> >>> soon. So users will have enough time to update their pipelines. There
> >>> are a couple of features missing to fully switch to DataStream API in
> >>> batch mode. Thanks for opening an issue, this is helpful for us to
> >>> gradually remove those barriers. They don't need to have a "Blocker"
> >>> priority in JIRA for now.
> >>
> >> Ok I thought the drop was sooner, no problem then.
> >>
> >>
> >>> But aggregations is a good example where we should discuss if it would
> >>> be easier to simply switch to Table API for that. Table API has a lot
> >>> of aggregation optimizations and can work on binary data. Also joins
> >>> should be easier in Table API. DataStream API can be a very low-level
> >>> API in the near future and most use cases (esp. the batch ones) should
> >>> be possible in Table API.
> >>
> >> Yes sure. As a matter of fact, my point was to use low level DataStream
> >> API in a benchmark to compare with Table API but I guess it is not a
> >> common user behavior.
> >>
> >>
> >>> Regarding: "Is it needed to port these Avro enhancements to new
> >>> DataStream connectors (add a new equivalent of
> >>> ParquetColumnarRowInputFormat but for Avro)"
> >>>
> >>> We should definitely not loose functionality. The same functionality
> >>> should be present in the new connectors. The questions is rather
> >>> whether we need to offer a DataStream API connector or if a Table API
> >>> connector would be nicer to use (also nicely integrated with catalogs).
> >>>
> >>> So a user can use a simple CREATE TABLE statement to configure the
> >>> connector; an easier abstraction is almost not possible. With
> >>> `tableEnv.toDataStream(table)` you can then continue in DataStream API
> >>> if there is still a need for it.
> >>
> >> Yes I agree, there is no easier connector setup than CREATE TABLE, and
> >> with tableEnv.toDataStream(table) if one would want to stay with
> >> DataStream (in my bench for example) it is still possible. And by the
> >> way the doc for parquet (1) for example only mentions using Table API
> >> connector.
> >>
> >> So I guess the table connector would be nicer for the user indeed. But
> >> if we use tableEnv.toDataStream(table), we would be able to produce only
> >> usual

Re: [DISCUSS] FLIP-182: Watermark alignment

[Seth Wiesman]

+1

In my opinion, this limitation is perfectly fine for the MVP. Watermark
alignment is a long-standing issue and this already moves the ball so far
forward.

I don't expect this will cause many issues in practice, as I understand it
the FileSource always processes one split at a time, and in my experience,
90% of Kafka users have a small number of partitions scale their pipelines
to have one reader per partition. Obviously, there are larger-scale Kafka
topics and more sources that will be ported over in the future but I think
there is an implicit understanding that aligning sources adds latency to
pipelines, and we can frame the follow-up "per-split" alignment as an
optimization.

On Fri, Jul 9, 2021 at 6:40 AM Piotr Nowojski 
wrote:

> Hey!
>
> A couple of weeks ago me and Arvid Heise played around with an idea to
> address a long standing issue of Flink: lack of watermark/event time
> alignment between different parallel instances of sources, that can lead to
> ever growing state size for downstream operators like WindowOperator.
>
> We had an impression that this is relatively low hanging fruit that can be
> quite easily implemented - at least partially (the first part mentioned in
> the FLIP document). I have written down our proposal [1] and you can also
> check out our PoC that we have implemented [2].
>
> We think that this is a quite easy proposal, that has been in large part
> already implemented. There is one obvious limitation of our PoC. Namely we
> can only easily block individual SourceOperators. This works perfectly fine
> as long as there is at most one split per SourceOperator. However it
> doesn't work with multiple splits. In that case, if a single
> `SourceOperator` is responsible for processing both the least and the most
> advanced splits, we won't be able to block this most advanced split for
> generating new records. I'm proposing to solve this problem in the future
> in another follow up FLIP, as a solution that works with a single split per
> operator is easier and already valuable for some of the users.
>
> What do you think about this proposal?
> Best, Piotrek
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources
> [2] https://github.com/pnowojski/flink/commits/aligned-sources
>

Re: [DISCUSS] Incrementally deprecating the DataSet API

[Seth Wiesman]

Hi Etienne,

The `toDataStream` method supports converting to concrete Java types, not
just Row, which can include your Avro specific-records. See example 2:

https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/data_stream_api/#examples-for-todatastream

On Thu, Jul 8, 2021 at 5:11 AM Etienne Chauchot 
wrote:

> Hi Timo,
>
> Thanks for your answers, no problem with the delay, I was in vacation
> too last week :)
>
> My comments are inline
>
> Best,
>
> Etienne
>
> On 07/07/2021 16:48, Timo Walther wrote:
> > Hi Etienne,
> >
> > sorry for the late reply due to my vacation last week.
> >
> > Regarding: "support of aggregations in batch mode for DataStream API
> > [...] is there a plan to solve it before the actual drop of DataSet API"
> >
> > Just to clarify it again: we will not drop the DataSet API any time
> > soon. So users will have enough time to update their pipelines. There
> > are a couple of features missing to fully switch to DataStream API in
> > batch mode. Thanks for opening an issue, this is helpful for us to
> > gradually remove those barriers. They don't need to have a "Blocker"
> > priority in JIRA for now.
>
>
> Ok I thought the drop was sooner, no problem then.
>
>
> >
> > But aggregations is a good example where we should discuss if it would
> > be easier to simply switch to Table API for that. Table API has a lot
> > of aggregation optimizations and can work on binary data. Also joins
> > should be easier in Table API. DataStream API can be a very low-level
> > API in the near future and most use cases (esp. the batch ones) should
> > be possible in Table API.
>
>
> Yes sure. As a matter of fact, my point was to use low level DataStream
> API in a benchmark to compare with Table API but I guess it is not a
> common user behavior.
>
>
> >
> > Regarding: "Is it needed to port these Avro enhancements to new
> > DataStream connectors (add a new equivalent of
> > ParquetColumnarRowInputFormat but for Avro)"
> >
> > We should definitely not loose functionality. The same functionality
> > should be present in the new connectors. The questions is rather
> > whether we need to offer a DataStream API connector or if a Table API
> > connector would be nicer to use (also nicely integrated with catalogs).
> >
> > So a user can use a simple CREATE TABLE statement to configure the
> > connector; an easier abstraction is almost not possible. With
> > `tableEnv.toDataStream(table)` you can then continue in DataStream API
> > if there is still a need for it.
>
>
> Yes I agree, there is no easier connector setup than CREATE TABLE, and
> with tableEnv.toDataStream(table) if one would want to stay with
> DataStream (in my bench for example) it is still possible. And by the
> way the doc for parquet (1) for example only mentions using Table API
> connector.
>
> So I guess the table connector would be nicer for the user indeed. But
> if we use tableEnv.toDataStream(table), we would be able to produce only
> usual types like Row, Tuples or Pojos, we still need to add Avro support
> right ?
>
> [1]
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/formats/parquet/
>
>
> >
> > Regarding: "there are parquet bugs still open on deprecated parquet
> > connector"
> >
> > Yes, bugs should still be fixed in 1.13.
>
>
> OK
>
>
> >
> > Regarading: "I've been doing TPCDS benchmarks with Flink lately"
> >
> > Great to hear that :-)
>
>
> And congrats again on blink performances !
>
>
> >
> > Did you also see the recent discussion? A TPC-DS benchmark can further
> > be improved by providing statistics. Maybe this is helpful to you:
> >
> >
> https://lists.apache.org/thread.html/ra383c23f230ab8e7fa16ec64b4f277c267d6358d55cc8a0edc77bb63%40%3Cuser.flink.apache.org%3E
> >
>
>
> No, I missed that thread, thanks for the pointer, I'll read it and
> comment if I have something to add.
>
>
> >
> > I will prepare a blog post shortly.
>
>
> Good to hear :)
>
>
> >
> > Regards,
> > Timo
> >
> >
> >
> > On 06.07.21 15:05, Etienne Chauchot wrote:
> >> Hi all,
> >>
> >> Any comments ?
> >>
> >> cheers,
> >>
> >> Etienne
> >>
> >> On 25/06/2021 15:09, Etienne Chauchot wrote:
> >>> Hi everyone,
> >>>
> >>> @Timo, my comments are inline for steps 2, 4 and 5, please tell me
> >>> what you think.
> >>>
> >>> Best
> >>>
> >>> Etienne
> >>>
> >>>
> >>> On 23/06/2021 15:27, Chesnay Schepler wrote:
>  If we want to publicize this plan more shouldn't we have a rough
>  timeline for when 2.0 is on the table?
> 
>  On 6/23/2021 2:44 PM, Stephan Ewen wrote:
> > Thanks for writing this up, this also reflects my understanding.
> >
> > I think a blog post would be nice, ideally with an explicit call for
> > feedback so we learn about user concerns.
> > A blog post has a lot more reach than an ML thread.
> >
> > Best,
> > Stephan
> >
> >
> > On Wed, Jun 23, 2021 at 12:23 PM Timo Walther 
> > wrote:
> >
> >> Hi everyone,
> >>

Re: [DISCUSS] Drop Mesos in 1.14

[Seth Wiesman]

Sorry if this is a naive question, I don't have any real Mesos experience.
Is it possible to deploy a standalone cluster on top of Mesos in the same
way you can with Kubernetes? If so, and there is still Mesos demand from
the community, we could document that process as the recommended deployment
mode going forward.

Seth

On Tue, Jun 22, 2021 at 5:02 AM Arvid Heise  wrote:

> +1 for dropping. Frankly speaking, I don't see it having any future (and
> D2iQ
> agrees).
>
> If there is a surprisingly huge demand, I'd try to evaluate plugins for it.
>
> On Tue, Jun 22, 2021 at 11:46 AM Till Rohrmann 
> wrote:
>
> > I'd be ok with dropping support for Mesos if it helps us to clear our
> > dependencies in the flink-runtime module. If we do it, then we should
> > probably update our documentation with a pointer to the latest Flink
> > version that supports Mesos in case of users strictly need Mesos.
> >
> > Cheers,
> > Till
> >
> > On Tue, Jun 22, 2021 at 10:29 AM Chesnay Schepler 
> > wrote:
> >
> > > Last week I spent some time looking into making flink-runtime scala
> > > free, which effectively means to move the Akka-reliant classes to
> > > another module, and load that module along with Akka and all of it's
> > > dependencies (including Scala) through a separate classloader.
> > >
> > > This would finally decouple the Scala versions required by the runtime
> > > and API, and would allow us to upgrade Akka as we'd no longer be
> limited
> > > to Scala 2.11. It would rid the classpath of a few dependencies, and
> > > remove the need for scala suffixes on quite a few modules.
> > >
> > > However, our Mesos support has unfortunately a hard dependency on Akka,
> > > which naturally does not play well with the goal of isolating Akka in
> > > it's own ClassLoader.
> > >
> > > To solve this issue I was thinking of simple dropping flink-mesos in
> > > 1.14 (it was deprecated in 1.13).
> > >
> > > Truth be told, I picked this option because it is the easiest to do. We
> > > _could_ probably make things work somehow (likely by shipping a second
> > > Akka version just for flink-mesos), but it doesn't seem worth the
> hassle
> > > and would void some of the benefits. So far we kept flink-mesos around,
> > > despite not really developing it further, because it didn't hurt to
> have
> > > it in still in Flink, but this has now changed.
> > >
> > > Please tell me what you think.
> > >
> > >
> >
>

Re: [DISCUSS] Releasing Flink 1.11.4, 1.12.5, 1.13.2

[Seth Wiesman]

+1 to the release.

It would be great if we could get FLINK-23073 into 1.13.2. There's already
an open PR and it unblocks upgrading the table API walkthrough in
apache/flink-playgrounds to 1.13.

Seth

On Mon, Jun 21, 2021 at 6:28 AM Yun Tang  wrote:

> Hi Dawid,
>
> Thanks for driving this discussion, I am willing to volunteer as the
> release manager for these versions.
>
>
> Best
> Yun Tang
> 
> From: Konstantin Knauf 
> Sent: Friday, June 18, 2021 22:35
> To: dev 
> Subject: Re: [DISCUSS] Releasing Flink 1.11.4, 1.12.5, 1.13.2
>
> Hi Dawid,
>
> Thank you for starting the discussion. I'd like to add
> https://issues.apache.org/jira/browse/FLINK-23025 to the list for Flink
> 1.13.2.
>
> Cheers,
>
> Konstantin
>
> On Fri, Jun 18, 2021 at 3:26 PM Dawid Wysakowicz 
> wrote:
>
> > Hi devs,
> >
> > Quite recently we pushed, in our opinion, quite an important fix[1] for
> > unaligned checkpoints which disables UC for broadcast partitioning.
> > Without the fix there might be some broadcast state corruption.
> > Therefore we think it would be beneficial to release it soonish. What do
> > you think? Do you have other issues in mind you'd like to have included
> > in these versions.
> >
> > Would someone be willing to volunteer to help with the releases as a
> > release manager? I guess there is a couple of spots to fill in here ;)
> >
> > Best,
> >
> > Dawid
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-22815
> >
> >
> >
>
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk
>

[jira] [Created] (FLINK-23073) Fix space handling in CSV timestamp parser

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-23073:


 Summary: Fix space handling in CSV timestamp parser
 Key: FLINK-23073
 URL: https://issues.apache.org/jira/browse/FLINK-23073
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.14.0, 1.13.2
Reporter: Seth Wiesman
Assignee: Seth Wiesman


FLINK-21947 Added support for TIMESTAMP_LTZ in the CSV format by replacing 
java.sql.Timestamp.valueOf with java.time.LocalDateTime.parse. 
Timestamp.valueOf internally calls `trim()` on the string before parsing while 
LocalDateTime.parse does not. This caused a breaking change where the CSV 
format can no longer parse timestamps of CSV's with spaces after the delimiter. 
We should manually re-add the call to trim to revert the behavior.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22947) Reduce DataSet visibility in documentation

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-22947:


 Summary: Reduce DataSet visibility in documentation
 Key: FLINK-22947
 URL: https://issues.apache.org/jira/browse/FLINK-22947
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Seth Wiesman
Assignee: Seth Wiesman
 Fix For: 1.14.0


Now that the legacy planner has been dropped and bounded datastream execution 
has landed, the primary remaining use cases for DataSet are gone however it is 
still featured prominently in our documentation.

 

In particular, it is listed second under Application Development, below 
DataStream but above table and Python. Additionally, Application Development 
has top-level dropdowns for serialization and managing execution even though 
these sections are only relevant for JVM DataStream and DataSet.

 

We should

1) move dataset to the bottom of the list and add a legacy label

2) move serialization to be a subsection of datastream

3) evaluate which pages under managing execution are still relevant and see if 
there are more appropriate places to put them.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22906) Add build time to Flink documentation

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-22906:


 Summary: Add build time to Flink documentation
 Key: FLINK-22906
 URL: https://issues.apache.org/jira/browse/FLINK-22906
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Seth Wiesman
Assignee: Seth Wiesman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [Discuss] Planning Flink 1.14

[Seth Wiesman]

Hi Everyone,

+1 for the Release Managers. Thank you all for volunteering.

@Till Rohrmann  A common sentiment that I have heard
from many users is that upgrading off of 1.9 was very difficult. In
particular, a lot of people struggled to understand the new memory model.
Many users who required custom memory configurations in earlier versions
assumed they should carry those configurations into latter versions and
then found themselves with OOM and instability issues. The good news is
Flink did what it was supposed to do and so for the majority dropping their
custom configurations and just setting total process memory was the correct
solution; this was not an issue of a buggy release. The problem is people
do not read the release notes or fully understood the implications of the
change. Back to Kurt's point, this transition seems to have left a bad
taste in many mouths, slowing some user's adoption of newer versions. I
don't know I have a solution to this problem. I think it is more
communication than engineering, but I'm open to continuing the discussion.

On Thu, Jun 3, 2021 at 5:04 AM Till Rohrmann  wrote:

> Thanks for volunteering as our release managers Xintong, Dawid and Joe!
>
> Thanks for starting the discussion about the release date Kurt. Personally,
> I prefer in general shorter release cycles as it allows us to deliver
> features faster and people feel less pressure to merge half-done features
> last minute because they fear that they have to wait a long time if they
> missed the train. Also, it forces us to make the release process less of a
> stop-the-world event and cut down the costs of releases.
>
> On the other hand, if our users don't upgrade Flink fast enough, then
> releasing more often won't have the effect of shipping features to our
> users and getting feedback faster from our users faster. What I believe we
> should try to do is to understand why upgrading Flink is so difficult for
> them. What are the things preventing a quick upgrade and how can we improve
> the situation for our users? Are our APIs not stable enough? Does Flink's
> behavior changes too drastically between versions? Is the tooling for
> upgrades lacking behind? Are they just cautious and don't want to use
> bleeding edge software?
>
> If there is a problem that the majority of users is using an unsupported
> version, then one solution could also be to extend the list of supported
> Flink versions to the latest 3 versions, for example.
>
> About your 2) point I am a bit skeptical. I think that we will simply plan
> more features and end up in the same situation wrt external contributions.
> If it weren't the case, then it would also work with shorter release cycles
> by simply planning less feature work and including the external
> contribution, which could not be done in the past release, in the next
> release. So in the end it is about what we plan for a release and not so
> much how much time we have (assuming that we plan less if we have less time
> and vice versa).
>
> Cheers,
> Till
>
> On Thu, Jun 3, 2021 at 5:08 AM Kurt Young  wrote:
>
> > Thanks for bringing this up.
> >
> > I have one thought about the release period. In a short word: shall we
> try
> > to extend the release period for 1 month?
> >
> > There are a couple of reasons why I want to bring up this proposal.
> >
> > 1) I observed that lots of users are actually far behind the current
> Flink
> > version. For example, we are now actively
> > developing 1.14 but most users I know who have a migration or upgrade
> plan
> > are planning to upgrade to 1.12. This means
> > we need to back port bug fixes to 1.12 and 1.13. If we extend the release
> > period by 1 month, I think there may be some
> > chances that users can have a proper time frame to upgrade to the
> previous
> > released version. Then we can have a
> > good development cycle which looks like "actively developing the current
> > version and making the previous version stable,
> > not 2 ~ 3 versions before". Always far away from Flink's latest version
> > also suppresses the motivation to contribute to Flink
> > from users perspective.
> >
> > 2) Increasing the release period also eases the workload of committers
> > which I think can improve the contributor experience.
> > I have seen several times that when some contributors want to do some new
> > features or improvements, we have to response
> > with "sorry we are right now focusing with implementing/stabilizing
> planned
> > feature for this version", and the contributions are
> > mostly like being stalled and never brought up again.
> >
> > BTW extending the release period also has downsides. It slows down the
> > delivery speed of new features. And I'm also not
> > sure how much it can improve the above 2 issues.
> >
> > Looking forward to hearing some feedback from the community, both users
> and
> > developers.
> >
> > Best,
> > Kurt
> >
> >
> > On Wed, Jun 2, 2021 at 8:39 PM JING ZHANG  wrote:
> >
> > > Hi Dawid, Joe & 

[jira] [Created] (FLINK-22563) Add migration guide for new StateBackend interfaces

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-22563:


 Summary: Add migration guide for new StateBackend interfaces
 Key: FLINK-22563
 URL: https://issues.apache.org/jira/browse/FLINK-22563
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.14.0, 1.13.1
Reporter: Seth Wiesman
Assignee: Seth Wiesman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22554) Support Kafka Topic Patterns in Kafka Ingress

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-22554:


 Summary: Support Kafka Topic Patterns in Kafka Ingress
 Key: FLINK-22554
 URL: https://issues.apache.org/jira/browse/FLINK-22554
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Affects Versions: statefun-3.1.0
Reporter: Seth Wiesman


Flink's Kafka source supports subscription patterns, where it will consume all 
topics that match a specified regex and periodically monitor for new topics. We 
should support this in the statefun Kafka ingress as it is generally useful and 
would remove a source of cluster downtime (subscribing to a new topic).

 

I propose something like this.

 
{code:java}
topics:
- topic-pattern: my-topic-*
  valueType: blah
  targets: 
  - blah{code}
The Flink consumer can be configured with both a list of concrete topics + a 
pattern so validation is simple.

 

cc [~igalshilman]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22242) Improve missing attribute error on GeneratedAddressedScopedStorage

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-22242:


 Summary: Improve missing attribute error on 
GeneratedAddressedScopedStorage
 Key: FLINK-22242
 URL: https://issues.apache.org/jira/browse/FLINK-22242
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Seth Wiesman
Assignee: Seth Wiesman
 Fix For: statefun-3.1.0


On the statefun python API, if a user mistypes a storage field name, or does 
not register a type, the storage class throws an unhelpful attribute error.

 

We can add a more helpful error message that does not expose internal classes 
and guides users towards a solution.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Apache Flink Stateful Functions 3.0.0, release candidate #1

[Seth Wiesman]

+1 (non-binding)

- Built from source and executed end to end tests
- Checked licenses and signatures
- Deployed remote Java SDK to gke cluster
- Took savepoint and statefully rescaled

Seth

On Thu, Apr 1, 2021 at 9:05 AM Konstantin Knauf  wrote:

> +1 (non-binding)
>
> - mvn clean install -Prun-e2e-tests (java 8) from source
> - python3 -m unittest tests
> - spin up Statefun Cluster on EKS with an image built from the Dockerfiles
> of [1]
> - run Python & Java Greeter example on AWS Lambda
> - read through documentation (opened [2] to fix some tpoys)
>
> [1] https://github.com/apache/flink-statefun-docker/pull/13
> [2] https://github.com/apache/flink-statefun/pull/219
>
> On Thu, Apr 1, 2021 at 6:46 AM Tzu-Li (Gordon) Tai 
> wrote:
>
> > +1 (binding)
> >
> > - verified signatures and hashes
> > - NOTICE and LICENSE files in statefun-flink-distribution,
> > statefun-protobuf-shaded, and statefun-sdk-java looks sane
> > - maven clean install -Prun-e2e-tests (java 8) from source
> > - ran all examples and tutorials in apache/flink-statefun-playground with
> > the new artifacts
> > - Ran my SDK verifier utility [1] against the new Java and Python SDKs.
> >
> > Cheers,
> > Gordon
> >
> > [1] https://github.com/tzulitai/statefun-sdk-verifier
> >
> > On Wed, Mar 31, 2021 at 8:50 PM Igal Shilman  wrote:
> >
> > > Thanks Gordon for managing the release!
> > >
> > > +1 (non binding) from my side:
> > >
> > > Here are the results of my testing:
> > > - verified the signatures
> > > - verified that the source distribution doesn't contain any binary
> files
> > > - ran mvn clean install -Prun-e2e-tests with java8
> > > - ran the smoke test that sends 100 million messages locally.
> > > - extended the smoke test to include the remote sdks (1 function in the
> > > Java SDK, 1 function in the Python SDK), and it passes.
> > > - deployed to kubernetes with minio as an S3 replacement.
> > >
> > >
> > > On Tue, Mar 30, 2021 at 12:29 PM Tzu-Li (Gordon) Tai <
> > tzuli...@apache.org>
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release candidate #1 for the version
> > 3.0.0
> > > of
> > > > Apache Flink Stateful Functions, as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > **Testing Guideline**
> > > >
> > > > You can find here [1] a page in the project wiki on instructions for
> > > > testing.
> > > > To cast a vote, it is not necessary to perform all listed checks,
> > > > but please mention which checks you have performed when voting.
> > > >
> > > > **Release Overview**
> > > >
> > > > As an overview, the release consists of the following:
> > > > a) Stateful Functions canonical source distribution, to be deployed
> to
> > > the
> > > > release repository at dist.apache.org
> > > > b) Stateful Functions Python SDK distributions to be deployed to PyPI
> > > > c) Maven artifacts to be deployed to the Maven Central Repository
> > > > d) New Dockerfiles for the release
> > > >
> > > > **Staging Areas to Review**
> > > >
> > > > The staging areas containing the above mentioned artifacts are as
> > > follows,
> > > > for your review:
> > > > * All artifacts for a) and b) can be found in the corresponding dev
> > > > repository at dist.apache.org [2]
> > > > * All artifacts for c) can be found at the Apache Nexus Repository
> [3]
> > > >
> > > > All artifacts are signed with the key
> > > > 1C1E2394D3194E1944613488F320986D35C33D6A [4]
> > > >
> > > > Other links for your review:
> > > > * JIRA release notes [5]
> > > > * source code tag “release-3.0.0-rc1” [6]
> > > > * PR for the new Dockerfiles [7]
> > > >
> > > > **Vote Duration**
> > > >
> > > > The voting time will run for at least 72 hours. I’m targeting this
> vote
> > > to
> > > > last until April. 2nd, 12pm CET.
> > > > It is adopted by majority approval, with at least 3 PMC affirmative
> > > votes.
> > > >
> > > > Thanks,
> > > > Gordon
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release
> > > > [2]
> > > https://dist.apache.org/repos/dist/dev/flink/flink-statefun-3.0.0-rc1/
> > > > [3]
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1415/
> > > > [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [5]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348822
> > > > [6] https://github.com/apache/flink-statefun/tree/release-3.0.0-rc1
> > > > [7] https://github.com/apache/flink-statefun-docker/pull/13
> > > >
> > >
> >
>
>
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk
>

[jira] [Created] (FLINK-22052) Add FLIP-142 changes to PyFlink

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-22052:


 Summary: Add FLIP-142 changes to PyFlink
 Key: FLINK-22052
 URL: https://issues.apache.org/jira/browse/FLINK-22052
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Seth Wiesman
Assignee: Seth Wiesman
 Fix For: 1.13.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21724) Remove type annotation from python message.as_type

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-21724:


 Summary: Remove type annotation from python message.as_type
 Key: FLINK-21724
 URL: https://issues.apache.org/jira/browse/FLINK-21724
 Project: Flink
  Issue Type: Bug
  Components: Stateful Functions
Reporter: Seth Wiesman
Assignee: Seth Wiesman
 Fix For: statefun-3.0.0


message.as_type's return type is dynamic based on the given type parameter. It 
should not have a type annotation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21719) Update statefun developer docs for new protocol

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-21719:


 Summary: Update statefun developer docs for new protocol
 Key: FLINK-21719
 URL: https://issues.apache.org/jira/browse/FLINK-21719
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Stateful Functions
Reporter: Seth Wiesman
Assignee: Seth Wiesman
 Fix For: statefun-3.0.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-165: Operator's Flame Graphs

[Seth Wiesman]

Cool feature +1

There is a subsection called monitoring in the operations section of the
docs. It would fit nicely there.

Seth

On Tue, Mar 2, 2021 at 4:23 PM Alexander Fedulov 
wrote:

> Hi Piotr,
>
> Thanks for the comments - all valid points.
> We should definitely document how the Flame Graphs are constructed - I will
> work on the docs. Do you have a proposition about the part of which
> page/section they should become?
> I would like to also mention here that I plan to work on further
> improvements, such as the ability to "zoom in" into the Flame Graphs for
> the individual Tasks in the "unsquashed" form,  so some of those concerns
> should be mitigated in the future.
>
> Best,
>
> --
>
> Alexander Fedulov | Solutions Architect
>
> 
>
> Follow us @VervericaData
>
>
> On Tue, Mar 2, 2021 at 3:17 PM Piotr Nowojski 
> wrote:
>
> > Nice feature +1 from my side for it.
> >
> > In the PR I think we are missing documentation. I think it's especially
> > important to mention the limitations of this approach for performance
> > analysis. If we make it easy for the user to get such kind of data, it's
> > important they do not proverbially shoot themselves in their own foot
> with
> > false conclusions. We should clearly mention how those data are sampled,
> > and point to limitations such as:
> > - data from all threads/operators are squashed together, so if there is a
> > data skew it will be averaged out
> > - stack sampling is/can be biased (JVM threads are more likely to be
> > stopped in some places than others, while skipping/rarely stopping in the
> > true hot spots - so one should treat the results with a grain of salt
> below
> > a certain level)
> > - true bottleneck might be obscured by the fact flame graphs are
> squashing
> > results from all of the threads. There can be 60% of time spent in one
> > component according to a flame graph, but it might not necessarily be the
> > bottleneck, which could be in a completely different component running
> > which has a single thread burning 100% of a single CPU core, barely
> visible
> > in the flame graph at all.
> >
> > It's great to have such a nice tool readily and easily available, but we
> > need to make sure people who are using it are aware when it can be
> > misleading.
> >
> > Piotrek
> >
> > wt., 2 mar 2021 o 15:12 Till Rohrmann  napisał(a):
> >
> > > Ah ok. Thanks for the clarification Alex.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Tue, Mar 2, 2021 at 2:02 PM Alexander Fedulov <
> > alexan...@ververica.com>
> > > wrote:
> > >
> > > > It is passed back as part of the response to the asynchronous
> callback
> > > > within the coordinator and is used to decide if all outstanding
> > requests
> > > to
> > > > the parallel instances of a particular operator returned
> successfully.
> > If
> > > > so, the request is considered successful, sub-results are combined
> and
> > > the
> > > > thread info result future for an operator completes.
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/pull/15054/commits/281188a025077849efd630f1f7aa801ff79a9afd#diff-20a1c89043e8d480e7af6dd36596b3558be9c6e64f6f4cf065df97fe76411c50R150
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/pull/15054/commits/281188a025077849efd630f1f7aa801ff79a9afd#diff-20a1c89043e8d480e7af6dd36596b3558be9c6e64f6f4cf065df97fe76411c50R277
> > > >
> > > > Best,
> > > >
> > > > --
> > > >
> > > > Alexander Fedulov | Solutions Architect
> > > >
> > > > 
> > > >
> > > > Follow us @VervericaData
> > > >
> > > >
> > > > On Tue, Mar 2, 2021 at 12:28 PM Till Rohrmann 
> > > > wrote:
> > > >
> > > > > Why does the caller of TaskExecutorGateway.requestThreadInfoSamples
> > > need
> > > > to
> > > > > specify the request id? Is it because the caller can send a second
> > > > request
> > > > > with the same id? Or can the caller query the result of a previous
> > > > request
> > > > > by specifying the requestId?
> > > > >
> > > > > If the TaskExecutor does not need to know about the id, then we
> might
> > > be
> > > > > able to drop it.
> > > > >
> > > > > Cheers
> > > > > Till
> > > > >
> > > > > On Tue, Mar 2, 2021 at 9:42 AM Alexander Fedulov <
> > > > alexan...@ververica.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Till,
> > > > > >
> > > > > > Thanks for your comments.
> > > > > >
> > > > > > * What is the requestId used for in the RPC call?
> > > > > >
> > > > > > It is the handle that is used as the key in the
> > > > > > ThreadInfoRequestCoordinator's pending responses Map. I believe
> it
> > > was
> > > > > > called sampleId in the StackTraceSampleCoordinator, but I decided
> > to
> > > > > rename
> > > > > > it because there is also a ThreadInfoSampleService which is
> > actually
> > > > > > responsible for sampling the JVM numSamples number of times. I
> > found
> > > > that
> > > > > > the notion of what a sample is was a bit confusing. Now one
> thread
> > > info
> > > > > > request 

Re: [DISCUSS] Deprecation and removal of the legacy SQL planner

[Seth Wiesman]

Strong +1

Having two planners is confusing to users and the diverging semantics make
it difficult to provide useful learning material. It is time to rip the
bandage off.

Seth

On Fri, Feb 26, 2021 at 12:54 AM Kurt Young  wrote:

>  change.>
>
> Hi Timo,
>
> First of all I want to thank you for introducing this planner design back
> in 1.9, this is a great work
> that allows lots of blink features to be merged to Flink in a reasonably
> short time. It greatly
> accelerates the evolution speed of Table & SQL.
>
> Everything comes with a cost, as you said, right now we are facing the
> overhead of maintaining
> two planners and it causes bugs and also increases imbalance between these
> two planners. As
> a developer and also for the good of all Table & SQL users, I also think
> it's better for us to be more
> focused on a single planner.
>
> Your proposed roadmap looks good to me, +1 from my side and thanks
> again for all your efforts!
>
> Best,
> Kurt
>
>
> On Thu, Feb 25, 2021 at 5:01 PM Timo Walther  wrote:
>
> > Hi everyone,
> >
> > since Flink 1.9 we have supported two SQL planners. Most of the original
> > plan of FLIP-32 [1] has been implemented. The Blink code merge has been
> > completed and many additional features have been added exclusively to
> > the new planner. The new planner is now in a much better shape than the
> > legacy one.
> >
> > In order to avoid user confusion, reduce duplicate code, and improve
> > maintainability and testing times of the Flink project as a whole we
> > would like to propose the following steps to complete FLIP-32:
> >
> > In Flink 1.13:
> > - Deprecate the `flink-table-planner` module
> > - Deprecate `BatchTableEnvironment` for both Java, Scala, and Python
> >
> > In Flink 1.14:
> > - Drop `flink-table-planner` early
> > - Drop many deprecated interfaces and API on demand
> > - Rename `flink-table-planner-blink` to `flink-table-planner`
> > - Rename `flink-table-runtime-blink` to `flink-table-runtime`
> > - Remove references of "Blink" in the code base
> >
> > This will have an impact on users that still use DataSet API together
> > with Table API. With this change we will not support converting between
> > DataSet API and Table API anymore. We hope to compensate the missing
> > functionality in the new unified TableEnvironment and/or the batch mode
> > in DataStream API during 1.14 and 1.15. For this, we are looking for
> > further feedback which features are required in Table API/DataStream API
> > to have a smooth migration path.
> >
> > Looking forward to your feedback.
> >
> > Regards,
> > Timo
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
> >
>

[jira] [Created] (FLINK-21489) Hugo docs add two anchor links to headers

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-21489:


 Summary: Hugo docs add two anchor links to headers
 Key: FLINK-21489
 URL: https://issues.apache.org/jira/browse/FLINK-21489
 Project: Flink
  Issue Type: Bug
Reporter: Seth Wiesman
Assignee: Seth Wiesman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21422) Migrate StateFun docs to hugo

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-21422:


 Summary: Migrate StateFun docs to hugo 
 Key: FLINK-21422
 URL: https://issues.apache.org/jira/browse/FLINK-21422
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Stateful Functions
Reporter: Seth Wiesman
Assignee: Seth Wiesman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21384) Automatically copy maven dependencies to clipboard on click

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-21384:


 Summary: Automatically copy maven dependencies to clipboard on 
click 
 Key: FLINK-21384
 URL: https://issues.apache.org/jira/browse/FLINK-21384
 Project: Flink
  Issue Type: Improvement
Reporter: Seth Wiesman
Assignee: Seth Wiesman


Flink has a number of optional dependencies users may need to copy into their 
pom files to use, such as connectors and formats. The docs should automatically 
copy the maven dependency to the users' clipboard when clicked.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-164: Improve Schema Handling in Catalogs

[Seth Wiesman]

+1 (binding)

On Fri, Feb 12, 2021 at 8:25 AM Jark Wu  wrote:

> +1 (binding)
>
> Best,
> Jark
>
> > 2021年2月12日 20:37，Dawid Wysakowicz  写道：
> >
> > +1 (binding)
> >
> > Best,
> >
> > Dawid
> >
> > On 12/02/2021 13:33, Timo Walther wrote:
> >> Hi everyone,
> >>
> >> I'd like to start a vote on FLIP-164 [1] which was discussed in [2].
> >>
> >> The vote will be open for at least 72 hours. Unless there are any
> >> objections, I'll close it by February 17th, 2021 (due to weekend) if
> >> we have received sufficient votes.
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-164%3A+Improve+Schema+Handling+in+Catalogs
> >> [2]
> >>
> https://lists.apache.org/thread.html/r79064a448d38d5b3d091dfb4703ae3eb94d5c6dd3970f120be7d5b13%40%3Cdev.flink.apache.org%3E
> >>
> >> Regards,
> >> Timo
> >
>
>

[jira] [Created] (FLINK-21369) Document Checkpoint Storage

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-21369:


 Summary: Document Checkpoint Storage
 Key: FLINK-21369
 URL: https://issues.apache.org/jira/browse/FLINK-21369
 Project: Flink
  Issue Type: Improvement
Reporter: Seth Wiesman
Assignee: Seth Wiesman
 Fix For: 1.13.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21363) Fix baseurl in documentation

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-21363:


 Summary: Fix baseurl in documentation
 Key: FLINK-21363
 URL: https://issues.apache.org/jira/browse/FLINK-21363
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Reporter: Seth Wiesman
Assignee: Seth Wiesman
 Fix For: 1.13.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [Announce] Hugo migration complete

[Seth Wiesman]

Piotr,

docs/content is where Jekyll stores its build artifacts. In
release-1.11/release-1.12 it is hidden via .gitignore so its content is not
removed when switching to master from those branches. You only need to
remember to delete this folder when you are switching from release-1.11/12
to master AND you built the docs on that branch. Switching from master to
release-1.11/12 does not require anything.

Seth

On Wed, Feb 10, 2021 at 9:32 AM Piotr Nowojski  wrote:

> Hi Seth,
>
> To clarify if I understand it correctly. Does it mean that now every time
> we are switching between master/release-1.11/release-1.12 branches we will
> need to remember to remove docs/content?
>
> Best,
> Piotrek
>
> śr., 10 lut 2021 o 16:08 Seth Wiesman  napisał(a):
>
> > One thing at a time :)
> >
> > I'm creating tickets now for the ones on the PR but please create others
> as
> > they come up.
> >
> > On Wed, Feb 10, 2021 at 9:02 AM Chesnay Schepler 
> > wrote:
> >
> > > Thank you Seth for the work you put in. This is a really nice
> > improvement.
> > > (When do we migrate the Flink website? :P )
> > >
> > > Regarding the follow-ups raised in the PR, will you go over them and
> > > create tickets, or...?
> > >
> > > On 2/10/2021 3:40 PM, Seth Wiesman wrote:
> > > > Hi Everyone,
> > > >
> > > > I'm happy to say the migration of Flinks documentation to hugo is
> > > complete!
> > > > There will likely be some continued work over the next weeks to clean
> > up
> > > > the css but the documentation freeze is over.
> > > >
> > > > Please carefully read docs/README for details about how to build and
> > work
> > > > with our new documentation setup. I will be working to update the
> > > > documentation contribution guide on flink-web today.
> > > >
> > > > There were a lot of files that were automatically migrated from
> Jekyll
> > to
> > > > Hugo. If you notice an issue please create a ticket in Jira and tag
> me
> > > > (sjwiesman).
> > > >
> > > > If you have any questions please do not hesitate to ping me on a PR,
> I
> > > will
> > > > be extra vigilant to watch my Github notifications through this
> release
> > > > cycle.
> > > >
> > > > *IMPORTANT* Jekyll and Hugo both use a directory called docs/content
> > for
> > > > different purposes. Jekyll uses it to cache build artifacts and as
> such
> > > it
> > > > has been part of flinks .gitignore. This means the Jekyll content
> will
> > > *not*
> > > > be removed when you pull from master. Please manually delete these
> > files
> > > or
> > > > the site will fail to build. I strongly encourage you to delete
> > > docs/content
> > > > before pulling from master.
> > > >
> > > > Seth
> > > >
> > >
> > >
> >
>

Re: [Announce] Hugo migration complete

[Seth Wiesman]

One thing at a time :)

I'm creating tickets now for the ones on the PR but please create others as
they come up.

On Wed, Feb 10, 2021 at 9:02 AM Chesnay Schepler  wrote:

> Thank you Seth for the work you put in. This is a really nice improvement.
> (When do we migrate the Flink website? :P )
>
> Regarding the follow-ups raised in the PR, will you go over them and
> create tickets, or...?
>
> On 2/10/2021 3:40 PM, Seth Wiesman wrote:
> > Hi Everyone,
> >
> > I'm happy to say the migration of Flinks documentation to hugo is
> complete!
> > There will likely be some continued work over the next weeks to clean up
> > the css but the documentation freeze is over.
> >
> > Please carefully read docs/README for details about how to build and work
> > with our new documentation setup. I will be working to update the
> > documentation contribution guide on flink-web today.
> >
> > There were a lot of files that were automatically migrated from Jekyll to
> > Hugo. If you notice an issue please create a ticket in Jira and tag me
> > (sjwiesman).
> >
> > If you have any questions please do not hesitate to ping me on a PR, I
> will
> > be extra vigilant to watch my Github notifications through this release
> > cycle.
> >
> > *IMPORTANT* Jekyll and Hugo both use a directory called docs/content for
> > different purposes. Jekyll uses it to cache build artifacts and as such
> it
> > has been part of flinks .gitignore. This means the Jekyll content will
> *not*
> > be removed when you pull from master. Please manually delete these files
> or
> > the site will fail to build. I strongly encourage you to delete
> docs/content
> > before pulling from master.
> >
> > Seth
> >
>
>

[Announce] Hugo migration complete

[Seth Wiesman]

Hi Everyone,

I'm happy to say the migration of Flinks documentation to hugo is complete!
There will likely be some continued work over the next weeks to clean up
the css but the documentation freeze is over.

Please carefully read docs/README for details about how to build and work
with our new documentation setup. I will be working to update the
documentation contribution guide on flink-web today.

There were a lot of files that were automatically migrated from Jekyll to
Hugo. If you notice an issue please create a ticket in Jira and tag me
(sjwiesman).

If you have any questions please do not hesitate to ping me on a PR, I will
be extra vigilant to watch my Github notifications through this release
cycle.

*IMPORTANT* Jekyll and Hugo both use a directory called docs/content for
different purposes. Jekyll uses it to cache build artifacts and as such it
has been part of flinks .gitignore. This means the Jekyll content will *not*
be removed when you pull from master. Please manually delete these files or
the site will fail to build. I strongly encourage you to delete docs/content
before pulling from master.

Seth

[jira] [Created] (FLINK-21341) Update state reader to return InputFormat

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-21341:


 Summary: Update state reader to return InputFormat
 Key: FLINK-21341
 URL: https://issues.apache.org/jira/browse/FLINK-21341
 Project: Flink
  Issue Type: Improvement
  Components: API / State Processor
Affects Versions: 1.13.0
Reporter: Seth Wiesman


The state processor api on read currently takes an ExecutionEnvironment and 
returns a DataSet. As Flink now supports Bounded DataStream we want to support 
that as well but not have to maintain parallel apis. To that end we propose the 
following.

Introduce a new `load` method to Savepoint.

 
{code:java}
SavepointReader Savepoint#load(String path, StateBackend backend);
{code}
SavepointReader will contain the same read methods as ExistingSavepoint but 
they will instead return InputFormat. This way the input format can be 
used with either DataSet, DataStream, or Table API.

 

The reader methods in ExistingSavepoint should be deprecated.

 

Additionally, OperatorStateInputFormat and KeyedStateInputFormat should both 
now extend ResultTypeQueryable so users get efficient serializer for their 
reads.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [Announce] Documentation Freeze Feb 2nd

[Seth Wiesman]

Reminder

On Thu, Jan 28, 2021 at 9:07 AM Seth Wiesman  wrote:

> Hi Everyone,
>
> As part of migrating the flink documentation to Hugo, I need to ask the
> community for a short documentation freeze.  This will keep us from losing
> any contributions during the migration. I am proposing the freeze begin
> next week February 2nd with the goal to get the change merged in that week.
> I have been working to have everything ready to go to keep this as
> unobtrusive as possible.
>
> If you have a pending documentation PR please do not rush it. If it is not
> merged before next Tuesday you will simply need to rebase after the
> migration is completed.
>
> Please let me know if you have any questions.
>
> Seth
>

[Announce] Documentation Freeze Feb 2nd

[Seth Wiesman]

Hi Everyone,

As part of migrating the flink documentation to Hugo, I need to ask the
community for a short documentation freeze.  This will keep us from losing
any contributions during the migration. I am proposing the freeze begin
next week February 2nd with the goal to get the change merged in that week.
I have been working to have everything ready to go to keep this as
unobtrusive as possible.

If you have a pending documentation PR please do not rush it. If it is not
merged before next Tuesday you will simply need to rebase after the
migration is completed.

Please let me know if you have any questions.

Seth

[jira] [Created] (FLINK-21193) Migrate Flink Docs to Hugo

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-21193:


 Summary: Migrate Flink Docs to Hugo 
 Key: FLINK-21193
 URL: https://issues.apache.org/jira/browse/FLINK-21193
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.13.0
Reporter: Seth Wiesman
Assignee: Seth Wiesman


Migrate the Flink documentation from Jekyll based to Hugo.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [Vote] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

[Seth Wiesman]

Hi all,

Looks like 157 has passed :)

I will open the tickets and move forward with the migration. Look out for a
follow-up email soon about the transition processes and short doc freeze.

Seth

On Wed, Jan 20, 2021 at 5:19 AM Arvid Heise  wrote:

> +1
>
> On Wed, Jan 20, 2021 at 12:07 PM Aljoscha Krettek 
> wrote:
>
> > +1
> >
> > Aljoscha
> >
> > On 2021/01/18 08:29, Seth Wiesman wrote:
> > >Hi devs,
> > >
> > >The discussion of the FLIP-157 [1] seems has reached a consensus through
> > >the mailing thread [2]. I would like to start a vote for it.
> > >
> > >The vote will be opened until 20th January (72h), unless there is an
> > >objection or no enough votes.
> > >
> > >Best,
> > >Seth
> > >
> > >[1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-157+Migrate+Flink+Documentation+from+Jekyll+to+Hugo
> > >[2]
> > >
> >
> https://lists.apache.org/thread.html/r88152bf178381c5e3bc2d7b3554cea3d61cff9ac0edb713dc518d9c7%40%3Cdev.flink.apache.org%3E
> >
>
>
> --
>
> Arvid Heise | Senior Java Developer
>
> <https://www.ververica.com/>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>

Re: Re: Re: [ANNOUNCE] Welcome Guowei Ma as a new Apache Flink Committer

[Seth Wiesman]

Congratulations!

On Wed, Jan 20, 2021 at 3:41 AM hailongwang <18868816...@163.com> wrote:

> Congratulations, Guowei!
>
> Best,
> Hailong
>
> 在 2021-01-20 15:55:24，"Till Rohrmann"  写道：
> >Congrats, Guowei!
> >
> >Cheers,
> >Till
> >
> >On Wed, Jan 20, 2021 at 8:32 AM Matthias Pohl 
> >wrote:
> >
> >> Congrats, Guowei!
> >>
> >> On Wed, Jan 20, 2021 at 8:22 AM Congxian Qiu 
> >> wrote:
> >>
> >> > Congrats Guowei!
> >> >
> >> > Best,
> >> > Congxian
> >> >
> >> >
> >> > Danny Chan  于2021年1月20日周三 下午2:59写道：
> >> >
> >> > > Congratulations Guowei!
> >> > >
> >> > > Best,
> >> > > Danny
> >> > >
> >> > > Jark Wu  于2021年1月20日周三 下午2:47写道：
> >> > >
> >> > > > Congratulations Guowei!
> >> > > >
> >> > > > Cheers,
> >> > > > Jark
> >> > > >
> >> > > > On Wed, 20 Jan 2021 at 14:36, SHI Xiaogang <
> shixiaoga...@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > > Congratulations MA!
> >> > > > >
> >> > > > > Regards,
> >> > > > > Xiaogang
> >> > > > >
> >> > > > > Yun Tang  于2021年1月20日周三 下午2:24写道：
> >> > > > >
> >> > > > > > Congratulations Guowei!
> >> > > > > >
> >> > > > > > Best
> >> > > > > > Yun Tang
> >> > > > > > 
> >> > > > > > From: Yang Wang 
> >> > > > > > Sent: Wednesday, January 20, 2021 13:59
> >> > > > > > To: dev 
> >> > > > > > Subject: Re: Re: [ANNOUNCE] Welcome Guowei Ma as a new Apache
> >> Flink
> >> > > > > > Committer
> >> > > > > >
> >> > > > > > Congratulations Guowei!
> >> > > > > >
> >> > > > > >
> >> > > > > > Best,
> >> > > > > > Yang
> >> > > > > >
> >> > > > > > Yun Gao  于2021年1月20日周三
> 下午1:52写道：
> >> > > > > >
> >> > > > > > > Congratulations Guowei!
> >> > > > > > >
> >> > > > > > > Best,
> >> > > > > > >
> >> > > >
> Yun--
> >> > > > > > > Sender:Yangze Guo
> >> > > > > > > Date:2021/01/20 13:48:52
> >> > > > > > > Recipient:dev
> >> > > > > > > Theme:Re: [ANNOUNCE] Welcome Guowei Ma as a new Apache Flink
> >> > > > Committer
> >> > > > > > >
> >> > > > > > > Congratulations, Guowei! Well deserved.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Best,
> >> > > > > > > Yangze Guo
> >> > > > > > >
> >> > > > > > > On Wed, Jan 20, 2021 at 1:46 PM Xintong Song <
> >> > > tonysong...@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > Congratulations, Guowei~!
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Thank you~
> >> > > > > > > >
> >> > > > > > > > Xintong Song
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Wed, Jan 20, 2021 at 1:42 PM Yuan Mei <
> >> > yuanmei.w...@gmail.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Congrats Guowei :-)
> >> > > > > > > > >
> >> > > > > > > > > Best,
> >> > > > > > > > > Yuan
> >> > > > > > > > >
> >> > > > > > > > > On Wed, Jan 20, 2021 at 1:36 PM tison <
> >> wander4...@gmail.com>
> >> > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Congrats Guowei!
> >> > > > > > > > > >
> >> > > > > > > > > > Best,
> >> > > > > > > > > > tison.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Kurt Young  于2021年1月20日周三 下午1:34写道：
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi everyone,
> >> > > > > > > > > > >
> >> > > > > > > > > > > I'm very happy to announce that Guowei Ma has
> accepted
> >> > the
> >> > > > > > > invitation
> >> > > > > > > > > to
> >> > > > > > > > > > > become a Flink committer.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Guowei is a very long term Flink developer, he has
> been
> >> > > > > extremely
> >> > > > > > > > > helpful
> >> > > > > > > > > > > with
> >> > > > > > > > > > > some important runtime changes, and also been
> active
> >> > with
> >> > > > > > > answering
> >> > > > > > > > > user
> >> > > > > > > > > > > questions as well as discussing designs.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Please join me in congratulating Guowei for
> becoming a
> >> > > Flink
> >> > > > > > > committer!
> >> > > > > > > > > > >
> >> > > > > > > > > > > Best,
> >> > > > > > > > > > > Kurt
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >>
>

[Announce] SQL docs are now Blink only

[Seth Wiesman]

Hi Everyone,

I just merged in a PR to make the SQL / Table docs Blink planner only.
Going forward, you do not need to mark something as Blink only or explain
divergent semantics. Simply write the docs as if Blink were the only
planner.

There is a Legacy planner specific page[1]. If you add a feature that is
not supported by the legacy planner, simply add a note to the legacy page
stating as such.

Seth

[1]
https://github.com/apache/flink/blob/master/docs/dev/table/legacy_planner.md

Re: [Vote] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

[Seth Wiesman]

Addendum, 72 hours from now is Thursday the 21st :)

sorry for the mistake.

Seth

On Mon, Jan 18, 2021 at 8:41 AM Timo Walther  wrote:

> +1
>
> Thanks for upgrading our docs infrastructure.
>
> Regards,
> Timo
>
> On 18.01.21 15:29, Seth Wiesman wrote:
> > Hi devs,
> >
> > The discussion of the FLIP-157 [1] seems has reached a consensus through
> > the mailing thread [2]. I would like to start a vote for it.
> >
> > The vote will be opened until 20th January (72h), unless there is an
> > objection or no enough votes.
> >
> > Best,
> > Seth
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-157+Migrate+Flink+Documentation+from+Jekyll+to+Hugo
> > [2]
> >
> https://lists.apache.org/thread.html/r88152bf178381c5e3bc2d7b3554cea3d61cff9ac0edb713dc518d9c7%40%3Cdev.flink.apache.org%3E
> >
>
>

[Vote] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

[Seth Wiesman]

Hi devs,

The discussion of the FLIP-157 [1] seems has reached a consensus through
the mailing thread [2]. I would like to start a vote for it.

The vote will be opened until 20th January (72h), unless there is an
objection or no enough votes.

Best,
Seth

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-157+Migrate+Flink+Documentation+from+Jekyll+to+Hugo
[2]
https://lists.apache.org/thread.html/r88152bf178381c5e3bc2d7b3554cea3d61cff9ac0edb713dc518d9c7%40%3Cdev.flink.apache.org%3E

[jira] [Created] (FLINK-20994) Add public method to create TableEnvironment in PyFlink

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-20994:


 Summary: Add public method to create TableEnvironment in PyFlink
 Key: FLINK-20994
 URL: https://issues.apache.org/jira/browse/FLINK-20994
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Affects Versions: 1.13.0
Reporter: Seth Wiesman


The Python table Api offers three table environment:
 * TableEnvironment
 * StreamTableEnvironment
 * BatchTableEnvironment

The `TableEnvironment` pydoc states that it a unified interface for pure table 
applications. How, it is currently missing a public constructor or `create` 
method to instantiate. This method should be added so users can leverage a 
unified interface and to better align pyflink with the JVM table apis.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: outdated version of com.esotericsoftware.kryo

[Seth Wiesman]

To expand on Till's answer, Flink uses Kryo whenever the type system does
not recognize a given type. If your job works with Kryo fallback disabled,
there should be no usage of Kryo serialization in your snapshots[1]. In
theory, it should be possible to upgrade Kryo using Flink's schema
migration capabilities but I'm not aware of anyone who has looked into the
details or is working on this.

Seth

[1]
https://ci.apache.org/projects/flink/flink-docs-master/dev/types_serialization.html#disabling-kryo-fallback

On Fri, Jan 15, 2021 at 10:46 AM Till Rohrmann  wrote:

> Yes, I think that Flink uses whatever key-/value-serializer is defined for
> serializing the keys/values for a savepoint.
>
> I haven't looked in detail into the problem but if Kryo offers some
> migration options, then this could make the problem a bit easier.
>
> Cheers,
> Till
>
>
>
> On Fri, Jan 15, 2021 at 2:44 PM Niels Basjes  wrote:
>
> > Till,
> >
> > So, we save the savepoints using the Kryo native format?
> > But I see that Kryo 2.24 (as mentioned here) is from may 2014 ...
> > that's ... really old !
> > https://github.com/EsotericSoftware/kryo/releases/tag/kryo-2.24.0
> >
> > I see that Kryo has some suggestions on the migration options:
> >
> >
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
> > One thing is that they have an extra artifact that allows having multiple
> > versions of Kryo in a single application to make this possible.
> >
> > Niels Basjes
> >
> > On Fri, Jan 15, 2021 at 2:14 PM Till Rohrmann 
> > wrote:
> >
> > > Hi Fred,
> > >
> > > one reason why this hasn't been updated is that Flink needs to ensure
> > > backwards compatibility. Hence, if the newer Kryo library serializes
> > > objects differently, then Flink would need to migrate old savepoints
> > having
> > > been stored via the old Kryo serializer to the new format. This is
> > > currently not supported and afaik blocks upgrading this library.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, Jan 15, 2021 at 11:41 AM Teunissen, F.G.J. (Fred)
> > >  wrote:
> > >
> > > > Hi,
> > > >
> > > > While I was resolving a dependency version conflict in our project
> that
> > > > uses Flink v1.11.2 I noticed that Flink is using the library
> > > > ‘com.esotericsoftware.kryo:kryo<
> > > > https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo
> > > > >:2.24.0’.
> > > > Is there a reason why this library isn’t updated to a more recent
> > version
> > > > (the last one is `com.esotericsoftware: kryo<
> > > > https://mvnrepository.com/artifact/com.esotericsoftware/kryo
> >:5.0.3`)
> > > >
> > > > Note. The groupId has been changed, so that could be the reason why
> it
> > is
> > > > missed.
> > > >
> > > > Regards,
> > > > Fred Teunissen
> > > >
> > > > -
> > > > ATTENTION:
> > > > The information in this e-mail is confidential and only meant for the
> > > > intended recipient. If you are not the intended recipient, don't use
> or
> > > > disclose it in any way. Please let the sender know and delete the
> > message
> > > > immediately.
> > > > -
> > > >
> > >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>

Re: [DISCUSS] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

[Seth Wiesman]

Great, if there aren't any other concerns I will open this up for a vote on
Monday.

Seth

On Thu, Jan 14, 2021 at 9:03 AM Seth Wiesman  wrote:

> Happy to see there is enthusiasm for this change, let me try and answers
> each of these questions.
>
> @Jark Wu  Hugo has proper support for i18n which means
> we can move content into external files that can be easily translated[1].
> For reference, Kubernetes has successfully used Hugo's built-in features to
> maintain 14 different languages[2]. Additionally, Hugo's md files are
> standard markdown which could allow us to integrate with other tooling. For
> example, we may look into using Crowdin for managing translations as the
> pulsar community does.
>
> @Till Rohrmann  None that I have found. In the
> proof of concept, I have already implemented all the Jekyll functionality
> we are using in the docs[4]. I have found Hugo shortcodes to be a more
> flexible alternative to liquid tags.
>
> @Chesnay Schepler  Not yet, I do not have access to
> the build bot (it is PMC only). I will work with INFRA to get Hugo
> installed if it is not already and Robert has agreed to set-up the build
> script on the build bot itself.
>
> Seth
>
> [1] https://gohugo.io/functions/i18n/
> [2] https://github.com/kubernetes/website/
> [3] https://github.com/apache/pulsar-translation
> [4]
> https://github.com/sjwiesman/flink-docs-v2/tree/master/layouts/shortcodes
>
>
>
> On Thu, Jan 14, 2021 at 7:03 AM David Anderson 
> wrote:
>
>> I've spent a few hours digging into this with Seth, and can report that
>> this makes working on the docs much less of a chore.
>>
>> +1 (with enthusiasm)
>>
>> Best,
>> David
>>
>> On Thu, Jan 14, 2021 at 1:34 PM Kostas Kloudas 
>> wrote:
>>
>> > +1 for moving to Hugo.
>> >
>> > Cheers,
>> > Kostas
>> >
>> > On Thu, Jan 14, 2021 at 1:27 PM Wei Zhong 
>> wrote:
>> > >
>> > > +1 for migrating to Hugo.
>> > >
>> > > Currently we have developed many plugins based on Jekyll because the
>> > native features of Jekyll cannot meet our needs. It seems all of them
>> can
>> > be supported via Hugo shortcodes and will become more concise.
>> > >
>> > > Best,
>> > > Wei
>> > >
>> > > > 在 2021年1月14日，18:21，Aljoscha Krettek  写道：
>> > > >
>> > > > +1
>> > > >
>> > > > The build times on Jekyll have just become to annoying for me. I
>> > realize that that is also a function of how we structure our
>> documentation,
>> > and especially how we construct the nav sidebar, but I think overall
>> moving
>> > to Hugo is still a benefit.
>> > > >
>> > > > Aljoscha
>> > > >
>> > > > On 2021/01/13 10:14, Seth Wiesman wrote:
>> > > >> Hi All,
>> > > >>
>> > > >> I would like to start a discussion for FLIP-157: Migrating the
>> Flink
>> > docs
>> > > >> from Jekyll to Hugo.
>> > > >>
>> > > >> This will allow us:
>> > > >>
>> > > >>  - Proper internationalization
>> > > >>  - Working Search
>> > > >>  - Sub-second build time ;)
>> > > >>
>> > > >> Please take a look and let me know what you think.
>> > > >>
>> > > >> Seth
>> > > >>
>> > > >>
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-157+Migrate+Flink+Documentation+from+Jekyll+to+Hugo
>> > >
>> >
>>
>

Re: [DISCUSS] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

[Seth Wiesman]

Happy to see there is enthusiasm for this change, let me try and answers
each of these questions.

@Jark Wu  Hugo has proper support for i18n which means we
can move content into external files that can be easily translated[1]. For
reference, Kubernetes has successfully used Hugo's built-in features to
maintain 14 different languages[2]. Additionally, Hugo's md files are
standard markdown which could allow us to integrate with other tooling. For
example, we may look into using Crowdin for managing translations as the
pulsar community does.

@Till Rohrmann  None that I have found. In the proof
of concept, I have already implemented all the Jekyll functionality we are
using in the docs[4]. I have found Hugo shortcodes to be a more flexible
alternative to liquid tags.

@Chesnay Schepler  Not yet, I do not have access to the
build bot (it is PMC only). I will work with INFRA to get Hugo installed if
it is not already and Robert has agreed to set-up the build script on the
build bot itself.

Seth

[1] https://gohugo.io/functions/i18n/
[2] https://github.com/kubernetes/website/
[3] https://github.com/apache/pulsar-translation
[4]
https://github.com/sjwiesman/flink-docs-v2/tree/master/layouts/shortcodes



On Thu, Jan 14, 2021 at 7:03 AM David Anderson  wrote:

> I've spent a few hours digging into this with Seth, and can report that
> this makes working on the docs much less of a chore.
>
> +1 (with enthusiasm)
>
> Best,
> David
>
> On Thu, Jan 14, 2021 at 1:34 PM Kostas Kloudas  wrote:
>
> > +1 for moving to Hugo.
> >
> > Cheers,
> > Kostas
> >
> > On Thu, Jan 14, 2021 at 1:27 PM Wei Zhong 
> wrote:
> > >
> > > +1 for migrating to Hugo.
> > >
> > > Currently we have developed many plugins based on Jekyll because the
> > native features of Jekyll cannot meet our needs. It seems all of them can
> > be supported via Hugo shortcodes and will become more concise.
> > >
> > > Best,
> > > Wei
> > >
> > > > 在 2021年1月14日，18:21，Aljoscha Krettek  写道：
> > > >
> > > > +1
> > > >
> > > > The build times on Jekyll have just become to annoying for me. I
> > realize that that is also a function of how we structure our
> documentation,
> > and especially how we construct the nav sidebar, but I think overall
> moving
> > to Hugo is still a benefit.
> > > >
> > > > Aljoscha
> > > >
> > > > On 2021/01/13 10:14, Seth Wiesman wrote:
> > > >> Hi All,
> > > >>
> > > >> I would like to start a discussion for FLIP-157: Migrating the Flink
> > docs
> > > >> from Jekyll to Hugo.
> > > >>
> > > >> This will allow us:
> > > >>
> > > >>  - Proper internationalization
> > > >>  - Working Search
> > > >>  - Sub-second build time ;)
> > > >>
> > > >> Please take a look and let me know what you think.
> > > >>
> > > >> Seth
> > > >>
> > > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-157+Migrate+Flink+Documentation+from+Jekyll+to+Hugo
> > >
> >
>

Re: [DISCUSS] Backport broadcast operations in BATCH mode to Flink

[Seth Wiesman]

+1

I would hope this helps attract more early adopters so if there are issues
we can resolve them in time for 1.13.

Seth

On Wed, Jan 13, 2021 at 5:13 AM Dawid Wysakowicz 
wrote:

> Hi,
>
> Given that the BATCH execution mode was only released in 1.12 and a
> rather small impact of the suggested change I'd be ok with backporting
> it to 1.12.x.
>
> Best,
>
> Dawid
>
> On 07/01/2021 12:50, Kostas Kloudas wrote:
> > +1 on my side as it does not break anything and it can act as motivation
> > for some people to upgrade.
> >
> > Cheers,
> > Kostas
> >
> > On Thu, 7 Jan 2021, 12:39 Aljoscha Krettek,  wrote:
> >
> >> 1.12.x
> >> Reply-To:
> >>
> >> Hi,
> >>
> >> what do you think about backporting FLINK-20491 [1] to Flink 1.12.x?
> >>
> >> I (we, including Dawid and Kostas) are a bit torn on this.
> >>
> >> a) It's a limitation of Flink 1.12.0 and fixing this seems very good for
> >> users that would otherwise have to wait until Flink 1.13.0.
> >>
> >> b) It's technically a new feature. We allow something with this change
> >> where previously an `UnsupportedOperationException` would be thrown.
> >>
> >> I would lean towards backporting this to 1.12.x. Thoughts?
> >>
> >> Best,
> >> Aljoscha
> >>
> >> [1] https://issues.apache.org/jira/browse/FLINK-20491
> >>
> >>
> >>
>
>

[DISCUSS] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

[Seth Wiesman]

Hi All,

I would like to start a discussion for FLIP-157: Migrating the Flink docs
from Jekyll to Hugo.

This will allow us:

   - Proper internationalization
   - Working Search
   - Sub-second build time ;)

Please take a look and let me know what you think.

Seth

https://cwiki.apache.org/confluence/display/FLINK/FLIP-157+Migrate+Flink+Documentation+from+Jekyll+to+Hugo

Re: [DISCUSS] FLIP-155: Introduce a few convenient operations in Table API

[Seth Wiesman]

This makes sense, I have some questions about method names.

What do you think about renaming `dropDuplicates` to `deduplicate`? I don't
think that drop is the right word to use for this operation, it implies
records are filtered where this operator actually issues updates and
retractions. Also, deduplicate is already how we talk about this feature in
the docs so I think it would be easier for users to find.

For null handling, I don't know how close we want to stick with SQL
conventions but what about making `coalesce` a top-level method? Something
like:

myTable.coalesce($("a"), 1).as("a")

We can require the next method to be an `as`. There is already precedent
for this sort of thing, `GroupedTable#aggregate` can only be followed by
`select`.

Seth

On Mon, Jan 4, 2021 at 6:27 AM Wei Zhong  wrote:

> Hi Dian,
>
> Big +1 for making the Table API easier to use. Java users and Python users
> can both benefit from it. I think it would be better if we add some Python
> API examples.
>
> Best,
> Wei
>
>
> > 在 2021年1月4日，20:03，Dian Fu  写道：
> >
> > Hi all,
> >
> > I'd like to start a discussion about introducing a few convenient
> operations in Table API from the perspective of ease of use.
> >
> > Currently some tasks are not easy to express in Table API e.g.
> deduplication, topn, etc, or not easy to express when there are hundreds of
> columns in a table, e.g. null data handling, etc.
> >
> > I'd like to propose to introduce a few operations in Table API with the
> following purposes:
> > - Make Table API users to easily leverage the powerful features already
> in SQL, e.g. deduplication, topn, etc
> > - Provide some convenient operations, e.g. introducing a series of
> operations for null data handling (it may become a problem when there are
> hundreds of columns), data sampling and splitting (which is a very common
> use case in ML which usually needs to split a table into multiple tables
> for training and validation separately).
> >
> > Please refer to FLIP-155 [1] for more details.
> >
> > Looking forward to your feedback!
> >
> > Regards,
> > Dian
> >
> > [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-155%3A+Introduce+a+few+convenient+operations+in+Table+API
>
>