RE: [DISCUSS] FLIP-212: Introduce Flink Kubernetes Operator

[zhengyu chen]

Hi, regarding the construction of k8s Flink Operator, I have already
completed some functions. I hope to contribute this part of the functions
and discuss with the community how to improve it. How should I start?

So far I have seen that the component has no operation persistence. Should
we persist its operation? for example, when I have a SessionCluster
deployment, I need to write its metadata to an external storage system in
yaml mode,
such as use mysql for storage. This design idea is similar to etcd in
k8s.If our  k8s Flink Operator application is restarted, We can recover
metadata information about deployment jobs, clusters, and so on based on
the database

Best
ConradJam

On 2022/01/25 05:08:01 Thomas Weise wrote:
> Hi,
>
> As promised in [1] we would like to start the discussion on the
> addition of a Kubernetes operator to the Flink project as FLIP-212:
>
https://cwiki.apache.org/confluence/display/FLINK/FLIP-212%3A+Introduce+Flink+Kubernetes+Operator
>
> Please note that the FLIP is currently focussed on the overall
> direction; the intention is to fill in more details once we converge
> on the high level plan.
>
> Thanks and looking forward to a lively discussion!
>
> Thomas
>
> [1] https://lists.apache.org/thread/l1dkp8v4bhlcyb4tdts99g7w4wdglfy4
>

Re: question about StatefunContext in golang Statefun SDK

[Galen Warren]

Hmm ... a downside to my proposal is that Go contexts are supposed to be
immutable, i.e. when adding a custom value to a context, a new context is
created with the new value and the old context isn't changed. Changing the
context.Context associated with the statefun.Context sort of goes against
the spirit of that, i.e. a consumer of statefun.Context could see custom
values change unexpectedly if another consumer of the same statefun.Context
modified the underlying context.Context.

To avoid that, I think we'd be back to having some mechanism to customize
the underlying context.Context once, when the statefun.Context is created
at the beginning of a stateful function invocation. Adding a field like:

PrepareContext func(ctx statefun.Context) context.Context

... to the StatefulFunctionSpec struct could accomplish that, i.e. if
PrepareContext were supplied, the context could be customized once at the
start of a function invocation and then left immutable after that point.

(Using statefun.Context as the input is deliberate here, in order to allow
the context.Context to be populated using values from the statefun.Context,
for example the function id).






On Fri, Feb 18, 2022 at 11:34 AM Galen Warren 
wrote:

> An example of passing it around would be:
>
> func (f *MyFunc) Invoke(ctx statefun.Context, message statefun.Message)
> error {
>
> logger := NewLogger()
> ctx.SetContext(ctxzap.ToContext(ctx, logger))
>
> return SomeOtherFunc(ctx)
> }
>
> func SomeOtherFunc(ctx context.Context) error {
>
> logger := ctxzap.Extract(ctx)
> logger.Info(...)
>
> return nil
> }
>
> This would also work with further nested calls, so long as the context is
> passed to them.
>
> On Fri, Feb 18, 2022 at 11:23 AM Galen Warren 
> wrote:
>
>> Ha, our emails keep passing.
>>
>> I've been playing around with options locally, and the SetContext option
>> seems to be the most flexible (and non-breaking), imo.
>>
>> The implementation would be trivial, just add:
>>
>> SetContext(ctx context.Context)
>>
>> ... to the statefun.Context interface, which is implemented as:
>>
>> func (s *statefunContext) SetContext(ctx context.Context) {
>> s.Context = ctx
>> }
>>
>>
>>
>>
>> On Fri, Feb 18, 2022 at 11:18 AM Austin Cawley-Edwards <
>> austin.caw...@gmail.com> wrote:
>>
>>> It would be helpful to have a small example though, if you have on Galen,
>>> to see how you're passing it around.
>>>
>>> On Fri, Feb 18, 2022 at 11:10 AM Austin Cawley-Edwards <
>>> austin.caw...@gmail.com> wrote:
>>>
>>> > Looking through the statefun Context interface, it indeed doesn't give
>>> > access to the underlying context.Context and the only implementation is
>>> > package-private [1]. I don't think there would be a way to update the
>>> > statfun.Context interface without introducing breaking changes, but if
>>> we
>>> > were to make that implementation public, that might be a stopgap
>>> solution.
>>> > e.g.,
>>> >
>>> > ```
>>> > type StatefunContext struct {
>>> > // expose embedded context
>>> > context.Context
>>> >
>>> > // make the mutext private
>>> > mu sync.Mutex
>>> >
>>> > // keep internals private
>>> > self Address
>>> > caller   *Address
>>> > storage  *storage
>>> > response *protocol.FromFunction_InvocationResponse
>>> > }
>>> > ```
>>> >
>>> > You could then do a type assertion in the handlers for this type of
>>> > context, and modify the context on it directly. It would be a bit
>>> ugly, but
>>> > may work.
>>> >
>>> > ```
>>> > func (s aFunc) Invoke(ctx Context, message Message) error {
>>> >   if sCtx, ok := ctx.(*statefun.StatefunContext); ok {
>>> > sCtx.Context = context.WithValue(sCtx.Context, "logger", aLogger)
>>> >   }
>>> >   // ...
>>> > }
>>> > ```
>>> >
>>> > Let me know what you all think,
>>> > Austin
>>> >
>>> >
>>> > [1]:
>>> >
>>> https://github.com/apache/flink-statefun/blob/1dfe226d85fea05a46c8ffa688175b4c0f2d4900/statefun-sdk-go/v3/pkg/statefun/context.go#L66-L73
>>> >
>>> >
>>> > On Fri, Feb 18, 2022 at 11:03 AM Galen Warren >> >
>>> > wrote:
>>> >
>>> >> Sorry Austin, I didn't see your response before I replied. Yes, we're
>>> >> saying the same thing.
>>> >>
>>> >> On Fri, Feb 18, 2022 at 10:56 AM Austin Cawley-Edwards <
>>> >> austin.caw...@gmail.com> wrote:
>>> >>
>>> >> > Hey all, jumping in. This makes sense to me – for instance to
>>> attach a
>>> >> > logger with some common metadata, e.g trace ID for the request?
>>> This is
>>> >> > common in go to add arbitrary items without updating the method
>>> >> signatures,
>>> >> > similar to thread local storage in Java.
>>> >> >
>>> >> > On Fri, Feb 18, 2022 at 10:53 AM Till Rohrmann <
>>> trohrm...@apache.org>
>>> >> > wrote:
>>> >> >
>>> >> > > Thanks for the clarification Galen. If you call the other Go
>>> >> functions,
>>> >> > > then you could also pass the other values as separate arguments to
>>> >> these
>>> >> > > functions, can't you?
>>> >> > >
>>> >> > > Cheers,
>>> >> > > Till
>>> >> > >
>>> >> > > On Fri, Feb 18, 

[jira] [Created] (FLINK-26257) Document metrics configuration for Prometheus

[Matyas Orhidi (Jira)]

Matyas Orhidi created FLINK-26257:
-

 Summary: Document metrics configuration for Prometheus
 Key: FLINK-26257
 URL: https://issues.apache.org/jira/browse/FLINK-26257
 Project: Flink
  Issue Type: Sub-task
Reporter: Matyas Orhidi






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: question about StatefunContext in golang Statefun SDK

[Galen Warren]

An example of passing it around would be:

func (f *MyFunc) Invoke(ctx statefun.Context, message statefun.Message)
error {

logger := NewLogger()
ctx.SetContext(ctxzap.ToContext(ctx, logger))

return SomeOtherFunc(ctx)
}

func SomeOtherFunc(ctx context.Context) error {

logger := ctxzap.Extract(ctx)
logger.Info(...)

return nil
}

This would also work with further nested calls, so long as the context is
passed to them.

On Fri, Feb 18, 2022 at 11:23 AM Galen Warren 
wrote:

> Ha, our emails keep passing.
>
> I've been playing around with options locally, and the SetContext option
> seems to be the most flexible (and non-breaking), imo.
>
> The implementation would be trivial, just add:
>
> SetContext(ctx context.Context)
>
> ... to the statefun.Context interface, which is implemented as:
>
> func (s *statefunContext) SetContext(ctx context.Context) {
> s.Context = ctx
> }
>
>
>
>
> On Fri, Feb 18, 2022 at 11:18 AM Austin Cawley-Edwards <
> austin.caw...@gmail.com> wrote:
>
>> It would be helpful to have a small example though, if you have on Galen,
>> to see how you're passing it around.
>>
>> On Fri, Feb 18, 2022 at 11:10 AM Austin Cawley-Edwards <
>> austin.caw...@gmail.com> wrote:
>>
>> > Looking through the statefun Context interface, it indeed doesn't give
>> > access to the underlying context.Context and the only implementation is
>> > package-private [1]. I don't think there would be a way to update the
>> > statfun.Context interface without introducing breaking changes, but if
>> we
>> > were to make that implementation public, that might be a stopgap
>> solution.
>> > e.g.,
>> >
>> > ```
>> > type StatefunContext struct {
>> > // expose embedded context
>> > context.Context
>> >
>> > // make the mutext private
>> > mu sync.Mutex
>> >
>> > // keep internals private
>> > self Address
>> > caller   *Address
>> > storage  *storage
>> > response *protocol.FromFunction_InvocationResponse
>> > }
>> > ```
>> >
>> > You could then do a type assertion in the handlers for this type of
>> > context, and modify the context on it directly. It would be a bit ugly,
>> but
>> > may work.
>> >
>> > ```
>> > func (s aFunc) Invoke(ctx Context, message Message) error {
>> >   if sCtx, ok := ctx.(*statefun.StatefunContext); ok {
>> > sCtx.Context = context.WithValue(sCtx.Context, "logger", aLogger)
>> >   }
>> >   // ...
>> > }
>> > ```
>> >
>> > Let me know what you all think,
>> > Austin
>> >
>> >
>> > [1]:
>> >
>> https://github.com/apache/flink-statefun/blob/1dfe226d85fea05a46c8ffa688175b4c0f2d4900/statefun-sdk-go/v3/pkg/statefun/context.go#L66-L73
>> >
>> >
>> > On Fri, Feb 18, 2022 at 11:03 AM Galen Warren 
>> > wrote:
>> >
>> >> Sorry Austin, I didn't see your response before I replied. Yes, we're
>> >> saying the same thing.
>> >>
>> >> On Fri, Feb 18, 2022 at 10:56 AM Austin Cawley-Edwards <
>> >> austin.caw...@gmail.com> wrote:
>> >>
>> >> > Hey all, jumping in. This makes sense to me – for instance to attach
>> a
>> >> > logger with some common metadata, e.g trace ID for the request? This
>> is
>> >> > common in go to add arbitrary items without updating the method
>> >> signatures,
>> >> > similar to thread local storage in Java.
>> >> >
>> >> > On Fri, Feb 18, 2022 at 10:53 AM Till Rohrmann > >
>> >> > wrote:
>> >> >
>> >> > > Thanks for the clarification Galen. If you call the other Go
>> >> functions,
>> >> > > then you could also pass the other values as separate arguments to
>> >> these
>> >> > > functions, can't you?
>> >> > >
>> >> > > Cheers,
>> >> > > Till
>> >> > >
>> >> > > On Fri, Feb 18, 2022 at 3:31 PM Galen Warren <
>> ga...@cvillewarrens.com
>> >> >
>> >> > > wrote:
>> >> > >
>> >> > > > The former.
>> >> > > >
>> >> > > > I think there's potential for confusion here because we're using
>> the
>> >> > > > word *function
>> >> > > > *in a couple of senses. One sense is a *stateful function*;
>> another
>> >> > sense
>> >> > > > is a *Go function*.
>> >> > > >
>> >> > > > What I'm looking to do is to put values in the Context so that
>> >> > downstream
>> >> > > > Go functions that receive the context can access those values.
>> Those
>> >> > > > downstream Go functions would be called during one invocation of
>> the
>> >> > > > stateful function.
>> >> > > >
>> >> > > > On Fri, Feb 18, 2022 at 6:48 AM Till Rohrmann <
>> trohrm...@apache.org
>> >> >
>> >> > > > wrote:
>> >> > > >
>> >> > > > > Hi Galen,
>> >> > > > >
>> >> > > > > Am I understanding it correctly, that you would like to set
>> some
>> >> > values
>> >> > > > in
>> >> > > > > the Context of function A that is then accessible in a
>> downstream
>> >> > call
>> >> > > of
>> >> > > > > function B? Or would you like to set a value that is accessible
>> >> once
>> >> > > > > function A is called again (w/ or w/o the same id)?
>> >> > > > >
>> >> > > > > Cheers,
>> >> > > > > Till
>> >> > > > >
>> >> > > > > On Thu, Feb 17, 2022 at 10:59 PM Galen Warren <
>> >> > ga...@cvillewarrens.com
>> >> > > >
>> 

Re: question about StatefunContext in golang Statefun SDK

[Galen Warren]

Ha, our emails keep passing.

I've been playing around with options locally, and the SetContext option
seems to be the most flexible (and non-breaking), imo.

The implementation would be trivial, just add:

SetContext(ctx context.Context)

... to the statefun.Context interface, which is implemented as:

func (s *statefunContext) SetContext(ctx context.Context) {
s.Context = ctx
}




On Fri, Feb 18, 2022 at 11:18 AM Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:

> It would be helpful to have a small example though, if you have on Galen,
> to see how you're passing it around.
>
> On Fri, Feb 18, 2022 at 11:10 AM Austin Cawley-Edwards <
> austin.caw...@gmail.com> wrote:
>
> > Looking through the statefun Context interface, it indeed doesn't give
> > access to the underlying context.Context and the only implementation is
> > package-private [1]. I don't think there would be a way to update the
> > statfun.Context interface without introducing breaking changes, but if we
> > were to make that implementation public, that might be a stopgap
> solution.
> > e.g.,
> >
> > ```
> > type StatefunContext struct {
> > // expose embedded context
> > context.Context
> >
> > // make the mutext private
> > mu sync.Mutex
> >
> > // keep internals private
> > self Address
> > caller   *Address
> > storage  *storage
> > response *protocol.FromFunction_InvocationResponse
> > }
> > ```
> >
> > You could then do a type assertion in the handlers for this type of
> > context, and modify the context on it directly. It would be a bit ugly,
> but
> > may work.
> >
> > ```
> > func (s aFunc) Invoke(ctx Context, message Message) error {
> >   if sCtx, ok := ctx.(*statefun.StatefunContext); ok {
> > sCtx.Context = context.WithValue(sCtx.Context, "logger", aLogger)
> >   }
> >   // ...
> > }
> > ```
> >
> > Let me know what you all think,
> > Austin
> >
> >
> > [1]:
> >
> https://github.com/apache/flink-statefun/blob/1dfe226d85fea05a46c8ffa688175b4c0f2d4900/statefun-sdk-go/v3/pkg/statefun/context.go#L66-L73
> >
> >
> > On Fri, Feb 18, 2022 at 11:03 AM Galen Warren 
> > wrote:
> >
> >> Sorry Austin, I didn't see your response before I replied. Yes, we're
> >> saying the same thing.
> >>
> >> On Fri, Feb 18, 2022 at 10:56 AM Austin Cawley-Edwards <
> >> austin.caw...@gmail.com> wrote:
> >>
> >> > Hey all, jumping in. This makes sense to me – for instance to attach a
> >> > logger with some common metadata, e.g trace ID for the request? This
> is
> >> > common in go to add arbitrary items without updating the method
> >> signatures,
> >> > similar to thread local storage in Java.
> >> >
> >> > On Fri, Feb 18, 2022 at 10:53 AM Till Rohrmann 
> >> > wrote:
> >> >
> >> > > Thanks for the clarification Galen. If you call the other Go
> >> functions,
> >> > > then you could also pass the other values as separate arguments to
> >> these
> >> > > functions, can't you?
> >> > >
> >> > > Cheers,
> >> > > Till
> >> > >
> >> > > On Fri, Feb 18, 2022 at 3:31 PM Galen Warren <
> ga...@cvillewarrens.com
> >> >
> >> > > wrote:
> >> > >
> >> > > > The former.
> >> > > >
> >> > > > I think there's potential for confusion here because we're using
> the
> >> > > > word *function
> >> > > > *in a couple of senses. One sense is a *stateful function*;
> another
> >> > sense
> >> > > > is a *Go function*.
> >> > > >
> >> > > > What I'm looking to do is to put values in the Context so that
> >> > downstream
> >> > > > Go functions that receive the context can access those values.
> Those
> >> > > > downstream Go functions would be called during one invocation of
> the
> >> > > > stateful function.
> >> > > >
> >> > > > On Fri, Feb 18, 2022 at 6:48 AM Till Rohrmann <
> trohrm...@apache.org
> >> >
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Galen,
> >> > > > >
> >> > > > > Am I understanding it correctly, that you would like to set some
> >> > values
> >> > > > in
> >> > > > > the Context of function A that is then accessible in a
> downstream
> >> > call
> >> > > of
> >> > > > > function B? Or would you like to set a value that is accessible
> >> once
> >> > > > > function A is called again (w/ or w/o the same id)?
> >> > > > >
> >> > > > > Cheers,
> >> > > > > Till
> >> > > > >
> >> > > > > On Thu, Feb 17, 2022 at 10:59 PM Galen Warren <
> >> > ga...@cvillewarrens.com
> >> > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Also, a potentially simpler way to support this would be to
> add
> >> a
> >> > > > > > SetContext method to the statefun.Context interface, and have
> it
> >> > > assign
> >> > > > > the
> >> > > > > > wrapped context. This would not require changes to the
> function
> >> > spec,
> >> > > > or
> >> > > > > > anything else, and would be more flexible.
> >> > > > > >
> >> > > > > > On Thu, Feb 17, 2022 at 1:05 PM Galen Warren <
> >> > > ga...@cvillewarrens.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Thanks for the quick reply!
> >> > > > > > >
> >> > > > > > > What I'm trying to do is put some things into the 

Re: question about StatefunContext in golang Statefun SDK

[Galen Warren]

Austin -- what would you think about an option to add SetContext(ctx
context.Context) as a method to the statefun.Context interface? Then one
could do something like this:

func (f *MyFunc) Invoke(ctx statefun.Context, message statefun.Message)
error {

logger := NewLogger()
ctx.SetContext(ctxzap.ToContext(ctx, logger))

}

... at any point that the context.Context internal to statefun.Context
needed to be updated? I think this would work with pretty much anything out
there that adds values to context.Context.

On Fri, Feb 18, 2022 at 11:12 AM Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:

> Looking through the statefun Context interface, it indeed doesn't give
> access to the underlying context.Context and the only implementation is
> package-private [1]. I don't think there would be a way to update the
> statfun.Context interface without introducing breaking changes, but if we
> were to make that implementation public, that might be a stopgap solution.
> e.g.,
>
> ```
> type StatefunContext struct {
> // expose embedded context
> context.Context
>
> // make the mutext private
> mu sync.Mutex
>
> // keep internals private
> self Address
> caller   *Address
> storage  *storage
> response *protocol.FromFunction_InvocationResponse
> }
> ```
>
> You could then do a type assertion in the handlers for this type of
> context, and modify the context on it directly. It would be a bit ugly, but
> may work.
>
> ```
> func (s aFunc) Invoke(ctx Context, message Message) error {
>   if sCtx, ok := ctx.(*statefun.StatefunContext); ok {
> sCtx.Context = context.WithValue(sCtx.Context, "logger", aLogger)
>   }
>   // ...
> }
> ```
>
> Let me know what you all think,
> Austin
>
>
> [1]:
>
> https://github.com/apache/flink-statefun/blob/1dfe226d85fea05a46c8ffa688175b4c0f2d4900/statefun-sdk-go/v3/pkg/statefun/context.go#L66-L73
>
>
> On Fri, Feb 18, 2022 at 11:03 AM Galen Warren 
> wrote:
>
> > Sorry Austin, I didn't see your response before I replied. Yes, we're
> > saying the same thing.
> >
> > On Fri, Feb 18, 2022 at 10:56 AM Austin Cawley-Edwards <
> > austin.caw...@gmail.com> wrote:
> >
> > > Hey all, jumping in. This makes sense to me – for instance to attach a
> > > logger with some common metadata, e.g trace ID for the request? This is
> > > common in go to add arbitrary items without updating the method
> > signatures,
> > > similar to thread local storage in Java.
> > >
> > > On Fri, Feb 18, 2022 at 10:53 AM Till Rohrmann 
> > > wrote:
> > >
> > > > Thanks for the clarification Galen. If you call the other Go
> functions,
> > > > then you could also pass the other values as separate arguments to
> > these
> > > > functions, can't you?
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Fri, Feb 18, 2022 at 3:31 PM Galen Warren <
> ga...@cvillewarrens.com>
> > > > wrote:
> > > >
> > > > > The former.
> > > > >
> > > > > I think there's potential for confusion here because we're using
> the
> > > > > word *function
> > > > > *in a couple of senses. One sense is a *stateful function*; another
> > > sense
> > > > > is a *Go function*.
> > > > >
> > > > > What I'm looking to do is to put values in the Context so that
> > > downstream
> > > > > Go functions that receive the context can access those values.
> Those
> > > > > downstream Go functions would be called during one invocation of
> the
> > > > > stateful function.
> > > > >
> > > > > On Fri, Feb 18, 2022 at 6:48 AM Till Rohrmann <
> trohrm...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Galen,
> > > > > >
> > > > > > Am I understanding it correctly, that you would like to set some
> > > values
> > > > > in
> > > > > > the Context of function A that is then accessible in a downstream
> > > call
> > > > of
> > > > > > function B? Or would you like to set a value that is accessible
> > once
> > > > > > function A is called again (w/ or w/o the same id)?
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Thu, Feb 17, 2022 at 10:59 PM Galen Warren <
> > > ga...@cvillewarrens.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Also, a potentially simpler way to support this would be to
> add a
> > > > > > > SetContext method to the statefun.Context interface, and have
> it
> > > > assign
> > > > > > the
> > > > > > > wrapped context. This would not require changes to the function
> > > spec,
> > > > > or
> > > > > > > anything else, and would be more flexible.
> > > > > > >
> > > > > > > On Thu, Feb 17, 2022 at 1:05 PM Galen Warren <
> > > > ga...@cvillewarrens.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for the quick reply!
> > > > > > > >
> > > > > > > > What I'm trying to do is put some things into the context so
> > that
> > > > > > they're
> > > > > > > > available in downstream calls, perhaps in methods with
> pointer
> > > > > > receivers
> > > > > > > to
> > > > > > > > the function struct (MyFunc) but also perhaps in methods that
> > are
> > > > > > further
> > > > > > 

Re: question about StatefunContext in golang Statefun SDK

[Austin Cawley-Edwards]

It would be helpful to have a small example though, if you have on Galen,
to see how you're passing it around.

On Fri, Feb 18, 2022 at 11:10 AM Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:

> Looking through the statefun Context interface, it indeed doesn't give
> access to the underlying context.Context and the only implementation is
> package-private [1]. I don't think there would be a way to update the
> statfun.Context interface without introducing breaking changes, but if we
> were to make that implementation public, that might be a stopgap solution.
> e.g.,
>
> ```
> type StatefunContext struct {
> // expose embedded context
> context.Context
>
> // make the mutext private
> mu sync.Mutex
>
> // keep internals private
> self Address
> caller   *Address
> storage  *storage
> response *protocol.FromFunction_InvocationResponse
> }
> ```
>
> You could then do a type assertion in the handlers for this type of
> context, and modify the context on it directly. It would be a bit ugly, but
> may work.
>
> ```
> func (s aFunc) Invoke(ctx Context, message Message) error {
>   if sCtx, ok := ctx.(*statefun.StatefunContext); ok {
> sCtx.Context = context.WithValue(sCtx.Context, "logger", aLogger)
>   }
>   // ...
> }
> ```
>
> Let me know what you all think,
> Austin
>
>
> [1]:
> https://github.com/apache/flink-statefun/blob/1dfe226d85fea05a46c8ffa688175b4c0f2d4900/statefun-sdk-go/v3/pkg/statefun/context.go#L66-L73
>
>
> On Fri, Feb 18, 2022 at 11:03 AM Galen Warren 
> wrote:
>
>> Sorry Austin, I didn't see your response before I replied. Yes, we're
>> saying the same thing.
>>
>> On Fri, Feb 18, 2022 at 10:56 AM Austin Cawley-Edwards <
>> austin.caw...@gmail.com> wrote:
>>
>> > Hey all, jumping in. This makes sense to me – for instance to attach a
>> > logger with some common metadata, e.g trace ID for the request? This is
>> > common in go to add arbitrary items without updating the method
>> signatures,
>> > similar to thread local storage in Java.
>> >
>> > On Fri, Feb 18, 2022 at 10:53 AM Till Rohrmann 
>> > wrote:
>> >
>> > > Thanks for the clarification Galen. If you call the other Go
>> functions,
>> > > then you could also pass the other values as separate arguments to
>> these
>> > > functions, can't you?
>> > >
>> > > Cheers,
>> > > Till
>> > >
>> > > On Fri, Feb 18, 2022 at 3:31 PM Galen Warren > >
>> > > wrote:
>> > >
>> > > > The former.
>> > > >
>> > > > I think there's potential for confusion here because we're using the
>> > > > word *function
>> > > > *in a couple of senses. One sense is a *stateful function*; another
>> > sense
>> > > > is a *Go function*.
>> > > >
>> > > > What I'm looking to do is to put values in the Context so that
>> > downstream
>> > > > Go functions that receive the context can access those values. Those
>> > > > downstream Go functions would be called during one invocation of the
>> > > > stateful function.
>> > > >
>> > > > On Fri, Feb 18, 2022 at 6:48 AM Till Rohrmann > >
>> > > > wrote:
>> > > >
>> > > > > Hi Galen,
>> > > > >
>> > > > > Am I understanding it correctly, that you would like to set some
>> > values
>> > > > in
>> > > > > the Context of function A that is then accessible in a downstream
>> > call
>> > > of
>> > > > > function B? Or would you like to set a value that is accessible
>> once
>> > > > > function A is called again (w/ or w/o the same id)?
>> > > > >
>> > > > > Cheers,
>> > > > > Till
>> > > > >
>> > > > > On Thu, Feb 17, 2022 at 10:59 PM Galen Warren <
>> > ga...@cvillewarrens.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Also, a potentially simpler way to support this would be to add
>> a
>> > > > > > SetContext method to the statefun.Context interface, and have it
>> > > assign
>> > > > > the
>> > > > > > wrapped context. This would not require changes to the function
>> > spec,
>> > > > or
>> > > > > > anything else, and would be more flexible.
>> > > > > >
>> > > > > > On Thu, Feb 17, 2022 at 1:05 PM Galen Warren <
>> > > ga...@cvillewarrens.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Thanks for the quick reply!
>> > > > > > >
>> > > > > > > What I'm trying to do is put some things into the context so
>> that
>> > > > > they're
>> > > > > > > available in downstream calls, perhaps in methods with pointer
>> > > > > receivers
>> > > > > > to
>> > > > > > > the function struct (MyFunc) but also perhaps in methods that
>> are
>> > > > > further
>> > > > > > > downstream that don't have access to MyFunc. If I'm
>> understanding
>> > > > > > > correctly, your proposal would work for the former but not the
>> > > > latter.
>> > > > > > >
>> > > > > > > An example would be to put a configured Logger into the
>> context
>> > > via a
>> > > > > > > WithLogger method (logging package - knative.dev/pkg/logging
>> -
>> > > > > > pkg.go.dev
>> > > > > > > ) and
>> > then
>> > > > pull
>> > > > > > it
>> > > > > > > out downstream via FromContext (logging 

Re: question about StatefunContext in golang Statefun SDK

[Austin Cawley-Edwards]

Looking through the statefun Context interface, it indeed doesn't give
access to the underlying context.Context and the only implementation is
package-private [1]. I don't think there would be a way to update the
statfun.Context interface without introducing breaking changes, but if we
were to make that implementation public, that might be a stopgap solution.
e.g.,

```
type StatefunContext struct {
// expose embedded context
context.Context

// make the mutext private
mu sync.Mutex

// keep internals private
self Address
caller   *Address
storage  *storage
response *protocol.FromFunction_InvocationResponse
}
```

You could then do a type assertion in the handlers for this type of
context, and modify the context on it directly. It would be a bit ugly, but
may work.

```
func (s aFunc) Invoke(ctx Context, message Message) error {
  if sCtx, ok := ctx.(*statefun.StatefunContext); ok {
sCtx.Context = context.WithValue(sCtx.Context, "logger", aLogger)
  }
  // ...
}
```

Let me know what you all think,
Austin


[1]:
https://github.com/apache/flink-statefun/blob/1dfe226d85fea05a46c8ffa688175b4c0f2d4900/statefun-sdk-go/v3/pkg/statefun/context.go#L66-L73


On Fri, Feb 18, 2022 at 11:03 AM Galen Warren 
wrote:

> Sorry Austin, I didn't see your response before I replied. Yes, we're
> saying the same thing.
>
> On Fri, Feb 18, 2022 at 10:56 AM Austin Cawley-Edwards <
> austin.caw...@gmail.com> wrote:
>
> > Hey all, jumping in. This makes sense to me – for instance to attach a
> > logger with some common metadata, e.g trace ID for the request? This is
> > common in go to add arbitrary items without updating the method
> signatures,
> > similar to thread local storage in Java.
> >
> > On Fri, Feb 18, 2022 at 10:53 AM Till Rohrmann 
> > wrote:
> >
> > > Thanks for the clarification Galen. If you call the other Go functions,
> > > then you could also pass the other values as separate arguments to
> these
> > > functions, can't you?
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, Feb 18, 2022 at 3:31 PM Galen Warren 
> > > wrote:
> > >
> > > > The former.
> > > >
> > > > I think there's potential for confusion here because we're using the
> > > > word *function
> > > > *in a couple of senses. One sense is a *stateful function*; another
> > sense
> > > > is a *Go function*.
> > > >
> > > > What I'm looking to do is to put values in the Context so that
> > downstream
> > > > Go functions that receive the context can access those values. Those
> > > > downstream Go functions would be called during one invocation of the
> > > > stateful function.
> > > >
> > > > On Fri, Feb 18, 2022 at 6:48 AM Till Rohrmann 
> > > > wrote:
> > > >
> > > > > Hi Galen,
> > > > >
> > > > > Am I understanding it correctly, that you would like to set some
> > values
> > > > in
> > > > > the Context of function A that is then accessible in a downstream
> > call
> > > of
> > > > > function B? Or would you like to set a value that is accessible
> once
> > > > > function A is called again (w/ or w/o the same id)?
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Thu, Feb 17, 2022 at 10:59 PM Galen Warren <
> > ga...@cvillewarrens.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Also, a potentially simpler way to support this would be to add a
> > > > > > SetContext method to the statefun.Context interface, and have it
> > > assign
> > > > > the
> > > > > > wrapped context. This would not require changes to the function
> > spec,
> > > > or
> > > > > > anything else, and would be more flexible.
> > > > > >
> > > > > > On Thu, Feb 17, 2022 at 1:05 PM Galen Warren <
> > > ga...@cvillewarrens.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for the quick reply!
> > > > > > >
> > > > > > > What I'm trying to do is put some things into the context so
> that
> > > > > they're
> > > > > > > available in downstream calls, perhaps in methods with pointer
> > > > > receivers
> > > > > > to
> > > > > > > the function struct (MyFunc) but also perhaps in methods that
> are
> > > > > further
> > > > > > > downstream that don't have access to MyFunc. If I'm
> understanding
> > > > > > > correctly, your proposal would work for the former but not the
> > > > latter.
> > > > > > >
> > > > > > > An example would be to put a configured Logger into the context
> > > via a
> > > > > > > WithLogger method (logging package - knative.dev/pkg/logging -
> > > > > > pkg.go.dev
> > > > > > > ) and
> > then
> > > > pull
> > > > > > it
> > > > > > > out downstream via FromContext (logging package -
> > > > > > knative.dev/pkg/logging
> > > > > > > - pkg.go.dev <
> > > https://pkg.go.dev/knative.dev/pkg/logging#FromContext
> > > > > >).
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 16, 2022 at 5:50 PM Seth Wiesman <
> > sjwies...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Galen,
> > > > > > >>
> > > > > > >> No, that is not currently 

Re: question about StatefunContext in golang Statefun SDK

[Galen Warren]

Sorry Austin, I didn't see your response before I replied. Yes, we're
saying the same thing.

On Fri, Feb 18, 2022 at 10:56 AM Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:

> Hey all, jumping in. This makes sense to me – for instance to attach a
> logger with some common metadata, e.g trace ID for the request? This is
> common in go to add arbitrary items without updating the method signatures,
> similar to thread local storage in Java.
>
> On Fri, Feb 18, 2022 at 10:53 AM Till Rohrmann 
> wrote:
>
> > Thanks for the clarification Galen. If you call the other Go functions,
> > then you could also pass the other values as separate arguments to these
> > functions, can't you?
> >
> > Cheers,
> > Till
> >
> > On Fri, Feb 18, 2022 at 3:31 PM Galen Warren 
> > wrote:
> >
> > > The former.
> > >
> > > I think there's potential for confusion here because we're using the
> > > word *function
> > > *in a couple of senses. One sense is a *stateful function*; another
> sense
> > > is a *Go function*.
> > >
> > > What I'm looking to do is to put values in the Context so that
> downstream
> > > Go functions that receive the context can access those values. Those
> > > downstream Go functions would be called during one invocation of the
> > > stateful function.
> > >
> > > On Fri, Feb 18, 2022 at 6:48 AM Till Rohrmann 
> > > wrote:
> > >
> > > > Hi Galen,
> > > >
> > > > Am I understanding it correctly, that you would like to set some
> values
> > > in
> > > > the Context of function A that is then accessible in a downstream
> call
> > of
> > > > function B? Or would you like to set a value that is accessible once
> > > > function A is called again (w/ or w/o the same id)?
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Thu, Feb 17, 2022 at 10:59 PM Galen Warren <
> ga...@cvillewarrens.com
> > >
> > > > wrote:
> > > >
> > > > > Also, a potentially simpler way to support this would be to add a
> > > > > SetContext method to the statefun.Context interface, and have it
> > assign
> > > > the
> > > > > wrapped context. This would not require changes to the function
> spec,
> > > or
> > > > > anything else, and would be more flexible.
> > > > >
> > > > > On Thu, Feb 17, 2022 at 1:05 PM Galen Warren <
> > ga...@cvillewarrens.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks for the quick reply!
> > > > > >
> > > > > > What I'm trying to do is put some things into the context so that
> > > > they're
> > > > > > available in downstream calls, perhaps in methods with pointer
> > > > receivers
> > > > > to
> > > > > > the function struct (MyFunc) but also perhaps in methods that are
> > > > further
> > > > > > downstream that don't have access to MyFunc. If I'm understanding
> > > > > > correctly, your proposal would work for the former but not the
> > > latter.
> > > > > >
> > > > > > An example would be to put a configured Logger into the context
> > via a
> > > > > > WithLogger method (logging package - knative.dev/pkg/logging -
> > > > > pkg.go.dev
> > > > > > ) and
> then
> > > pull
> > > > > it
> > > > > > out downstream via FromContext (logging package -
> > > > > knative.dev/pkg/logging
> > > > > > - pkg.go.dev <
> > https://pkg.go.dev/knative.dev/pkg/logging#FromContext
> > > > >).
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 16, 2022 at 5:50 PM Seth Wiesman <
> sjwies...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Hi Galen,
> > > > > >>
> > > > > >> No, that is not currently supported, the current idiomatic way
> > would
> > > > be
> > > > > to
> > > > > >> pass those values to the struct implementing the Statefun
> > interface.
> > > > > >>
> > > > > >>
> > > > > >> type MyFunc struct { someRuntimeInfo string } func (m *MyFunc)
> > > > > Invoke(ctx
> > > > > >> statefun.Context, message statefun.Message) error { } func
> main()
> > {
> > > > > >> builder
> > > > > >> := statefun.StatefulFunctionsBuilder()
> > > > > >> f := MyFunc { someRuntimeInfo: "runtime-provided" }
> > builder.WithSpec
> > > > > >> (statefun.StatefulFunctionSpec{ FunctionType:
> > statefun.TypeNameFrom(
> > > > > >> "example/my-func"), Function: f })
> > > > > >> http.Handle("/statefun", builder.AsHandler())
> > > > > >> _ = http.ListenAndServe(":8000", nil) }
> > > > > >>
> > > > > >> Would this work for you? Or what is the context (pun intended)
> you
> > > are
> > > > > >> looking for?
> > > > > >>
> > > > > >> Seth
> > > > > >>
> > > > > >> On Wed, Feb 16, 2022 at 4:35 PM Galen Warren <
> > > ga...@cvillewarrens.com
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > When stateful functions are invoked, they are passed an
> instance
> > > of
> > > > > >> > statefun.Context, which wraps the context.Context received by
> > the
> > > > HTTP
> > > > > >> > request. Is there any way to customize that context.Context
> to,
> > > say,
> > > > > >> hold
> > > > > >> > custom values, using ctx.WithValue()? I don't see a way but I
> > > 

Re: question about StatefunContext in golang Statefun SDK

[Galen Warren]

Sure, you could do that. But it's a common pattern in Go to attach things
like loggers, request ids, etc. to the context so that they don't have to
be explicitly passed around. It would be convenient to be able to follow
this pattern with Statefun.

On Fri, Feb 18, 2022 at 10:53 AM Till Rohrmann  wrote:

> Thanks for the clarification Galen. If you call the other Go functions,
> then you could also pass the other values as separate arguments to these
> functions, can't you?
>
> Cheers,
> Till
>
> On Fri, Feb 18, 2022 at 3:31 PM Galen Warren 
> wrote:
>
> > The former.
> >
> > I think there's potential for confusion here because we're using the
> > word *function
> > *in a couple of senses. One sense is a *stateful function*; another sense
> > is a *Go function*.
> >
> > What I'm looking to do is to put values in the Context so that downstream
> > Go functions that receive the context can access those values. Those
> > downstream Go functions would be called during one invocation of the
> > stateful function.
> >
> > On Fri, Feb 18, 2022 at 6:48 AM Till Rohrmann 
> > wrote:
> >
> > > Hi Galen,
> > >
> > > Am I understanding it correctly, that you would like to set some values
> > in
> > > the Context of function A that is then accessible in a downstream call
> of
> > > function B? Or would you like to set a value that is accessible once
> > > function A is called again (w/ or w/o the same id)?
> > >
> > > Cheers,
> > > Till
> > >
> > > On Thu, Feb 17, 2022 at 10:59 PM Galen Warren  >
> > > wrote:
> > >
> > > > Also, a potentially simpler way to support this would be to add a
> > > > SetContext method to the statefun.Context interface, and have it
> assign
> > > the
> > > > wrapped context. This would not require changes to the function spec,
> > or
> > > > anything else, and would be more flexible.
> > > >
> > > > On Thu, Feb 17, 2022 at 1:05 PM Galen Warren <
> ga...@cvillewarrens.com>
> > > > wrote:
> > > >
> > > > > Thanks for the quick reply!
> > > > >
> > > > > What I'm trying to do is put some things into the context so that
> > > they're
> > > > > available in downstream calls, perhaps in methods with pointer
> > > receivers
> > > > to
> > > > > the function struct (MyFunc) but also perhaps in methods that are
> > > further
> > > > > downstream that don't have access to MyFunc. If I'm understanding
> > > > > correctly, your proposal would work for the former but not the
> > latter.
> > > > >
> > > > > An example would be to put a configured Logger into the context
> via a
> > > > > WithLogger method (logging package - knative.dev/pkg/logging -
> > > > pkg.go.dev
> > > > > ) and then
> > pull
> > > > it
> > > > > out downstream via FromContext (logging package -
> > > > knative.dev/pkg/logging
> > > > > - pkg.go.dev <
> https://pkg.go.dev/knative.dev/pkg/logging#FromContext
> > > >).
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Feb 16, 2022 at 5:50 PM Seth Wiesman 
> > > > wrote:
> > > > >
> > > > >> Hi Galen,
> > > > >>
> > > > >> No, that is not currently supported, the current idiomatic way
> would
> > > be
> > > > to
> > > > >> pass those values to the struct implementing the Statefun
> interface.
> > > > >>
> > > > >>
> > > > >> type MyFunc struct { someRuntimeInfo string } func (m *MyFunc)
> > > > Invoke(ctx
> > > > >> statefun.Context, message statefun.Message) error { } func main()
> {
> > > > >> builder
> > > > >> := statefun.StatefulFunctionsBuilder()
> > > > >> f := MyFunc { someRuntimeInfo: "runtime-provided" }
> builder.WithSpec
> > > > >> (statefun.StatefulFunctionSpec{ FunctionType:
> statefun.TypeNameFrom(
> > > > >> "example/my-func"), Function: f })
> > > > >> http.Handle("/statefun", builder.AsHandler())
> > > > >> _ = http.ListenAndServe(":8000", nil) }
> > > > >>
> > > > >> Would this work for you? Or what is the context (pun intended) you
> > are
> > > > >> looking for?
> > > > >>
> > > > >> Seth
> > > > >>
> > > > >> On Wed, Feb 16, 2022 at 4:35 PM Galen Warren <
> > ga...@cvillewarrens.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > When stateful functions are invoked, they are passed an instance
> > of
> > > > >> > statefun.Context, which wraps the context.Context received by
> the
> > > HTTP
> > > > >> > request. Is there any way to customize that context.Context to,
> > say,
> > > > >> hold
> > > > >> > custom values, using ctx.WithValue()? I don't see a way but I
> > wanted
> > > > to
> > > > >> > ask.
> > > > >> >
> > > > >> > If not, would you be interested in a PR to add this
> > functionality? A
> > > > >> simple
> > > > >> > way might be to add a property to StatefulFunctionSpec, say:
> > > > >> >
> > > > >> > TransformContext func(ctx context.Context) context.Context
> > > > >> >
> > > > >> > ... that, if supplied, would be called to create a customized
> > > context
> > > > >> that
> > > > >> > would be used downstream?
> > > > >> >
> > > > >> > Thanks.
> > > > >> >
> > > > >>
> > > > >
> > > 

Re: question about StatefunContext in golang Statefun SDK

[Austin Cawley-Edwards]

Hey all, jumping in. This makes sense to me – for instance to attach a
logger with some common metadata, e.g trace ID for the request? This is
common in go to add arbitrary items without updating the method signatures,
similar to thread local storage in Java.

On Fri, Feb 18, 2022 at 10:53 AM Till Rohrmann  wrote:

> Thanks for the clarification Galen. If you call the other Go functions,
> then you could also pass the other values as separate arguments to these
> functions, can't you?
>
> Cheers,
> Till
>
> On Fri, Feb 18, 2022 at 3:31 PM Galen Warren 
> wrote:
>
> > The former.
> >
> > I think there's potential for confusion here because we're using the
> > word *function
> > *in a couple of senses. One sense is a *stateful function*; another sense
> > is a *Go function*.
> >
> > What I'm looking to do is to put values in the Context so that downstream
> > Go functions that receive the context can access those values. Those
> > downstream Go functions would be called during one invocation of the
> > stateful function.
> >
> > On Fri, Feb 18, 2022 at 6:48 AM Till Rohrmann 
> > wrote:
> >
> > > Hi Galen,
> > >
> > > Am I understanding it correctly, that you would like to set some values
> > in
> > > the Context of function A that is then accessible in a downstream call
> of
> > > function B? Or would you like to set a value that is accessible once
> > > function A is called again (w/ or w/o the same id)?
> > >
> > > Cheers,
> > > Till
> > >
> > > On Thu, Feb 17, 2022 at 10:59 PM Galen Warren  >
> > > wrote:
> > >
> > > > Also, a potentially simpler way to support this would be to add a
> > > > SetContext method to the statefun.Context interface, and have it
> assign
> > > the
> > > > wrapped context. This would not require changes to the function spec,
> > or
> > > > anything else, and would be more flexible.
> > > >
> > > > On Thu, Feb 17, 2022 at 1:05 PM Galen Warren <
> ga...@cvillewarrens.com>
> > > > wrote:
> > > >
> > > > > Thanks for the quick reply!
> > > > >
> > > > > What I'm trying to do is put some things into the context so that
> > > they're
> > > > > available in downstream calls, perhaps in methods with pointer
> > > receivers
> > > > to
> > > > > the function struct (MyFunc) but also perhaps in methods that are
> > > further
> > > > > downstream that don't have access to MyFunc. If I'm understanding
> > > > > correctly, your proposal would work for the former but not the
> > latter.
> > > > >
> > > > > An example would be to put a configured Logger into the context
> via a
> > > > > WithLogger method (logging package - knative.dev/pkg/logging -
> > > > pkg.go.dev
> > > > > ) and then
> > pull
> > > > it
> > > > > out downstream via FromContext (logging package -
> > > > knative.dev/pkg/logging
> > > > > - pkg.go.dev <
> https://pkg.go.dev/knative.dev/pkg/logging#FromContext
> > > >).
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Feb 16, 2022 at 5:50 PM Seth Wiesman 
> > > > wrote:
> > > > >
> > > > >> Hi Galen,
> > > > >>
> > > > >> No, that is not currently supported, the current idiomatic way
> would
> > > be
> > > > to
> > > > >> pass those values to the struct implementing the Statefun
> interface.
> > > > >>
> > > > >>
> > > > >> type MyFunc struct { someRuntimeInfo string } func (m *MyFunc)
> > > > Invoke(ctx
> > > > >> statefun.Context, message statefun.Message) error { } func main()
> {
> > > > >> builder
> > > > >> := statefun.StatefulFunctionsBuilder()
> > > > >> f := MyFunc { someRuntimeInfo: "runtime-provided" }
> builder.WithSpec
> > > > >> (statefun.StatefulFunctionSpec{ FunctionType:
> statefun.TypeNameFrom(
> > > > >> "example/my-func"), Function: f })
> > > > >> http.Handle("/statefun", builder.AsHandler())
> > > > >> _ = http.ListenAndServe(":8000", nil) }
> > > > >>
> > > > >> Would this work for you? Or what is the context (pun intended) you
> > are
> > > > >> looking for?
> > > > >>
> > > > >> Seth
> > > > >>
> > > > >> On Wed, Feb 16, 2022 at 4:35 PM Galen Warren <
> > ga...@cvillewarrens.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > When stateful functions are invoked, they are passed an instance
> > of
> > > > >> > statefun.Context, which wraps the context.Context received by
> the
> > > HTTP
> > > > >> > request. Is there any way to customize that context.Context to,
> > say,
> > > > >> hold
> > > > >> > custom values, using ctx.WithValue()? I don't see a way but I
> > wanted
> > > > to
> > > > >> > ask.
> > > > >> >
> > > > >> > If not, would you be interested in a PR to add this
> > functionality? A
> > > > >> simple
> > > > >> > way might be to add a property to StatefulFunctionSpec, say:
> > > > >> >
> > > > >> > TransformContext func(ctx context.Context) context.Context
> > > > >> >
> > > > >> > ... that, if supplied, would be called to create a customized
> > > context
> > > > >> that
> > > > >> > would be used downstream?
> > > > >> >
> > > > >> > Thanks.
> > > > >> >
> > > > >>
> > > 

Re: question about StatefunContext in golang Statefun SDK

[Till Rohrmann]

Thanks for the clarification Galen. If you call the other Go functions,
then you could also pass the other values as separate arguments to these
functions, can't you?

Cheers,
Till

On Fri, Feb 18, 2022 at 3:31 PM Galen Warren 
wrote:

> The former.
>
> I think there's potential for confusion here because we're using the
> word *function
> *in a couple of senses. One sense is a *stateful function*; another sense
> is a *Go function*.
>
> What I'm looking to do is to put values in the Context so that downstream
> Go functions that receive the context can access those values. Those
> downstream Go functions would be called during one invocation of the
> stateful function.
>
> On Fri, Feb 18, 2022 at 6:48 AM Till Rohrmann 
> wrote:
>
> > Hi Galen,
> >
> > Am I understanding it correctly, that you would like to set some values
> in
> > the Context of function A that is then accessible in a downstream call of
> > function B? Or would you like to set a value that is accessible once
> > function A is called again (w/ or w/o the same id)?
> >
> > Cheers,
> > Till
> >
> > On Thu, Feb 17, 2022 at 10:59 PM Galen Warren 
> > wrote:
> >
> > > Also, a potentially simpler way to support this would be to add a
> > > SetContext method to the statefun.Context interface, and have it assign
> > the
> > > wrapped context. This would not require changes to the function spec,
> or
> > > anything else, and would be more flexible.
> > >
> > > On Thu, Feb 17, 2022 at 1:05 PM Galen Warren 
> > > wrote:
> > >
> > > > Thanks for the quick reply!
> > > >
> > > > What I'm trying to do is put some things into the context so that
> > they're
> > > > available in downstream calls, perhaps in methods with pointer
> > receivers
> > > to
> > > > the function struct (MyFunc) but also perhaps in methods that are
> > further
> > > > downstream that don't have access to MyFunc. If I'm understanding
> > > > correctly, your proposal would work for the former but not the
> latter.
> > > >
> > > > An example would be to put a configured Logger into the context via a
> > > > WithLogger method (logging package - knative.dev/pkg/logging -
> > > pkg.go.dev
> > > > ) and then
> pull
> > > it
> > > > out downstream via FromContext (logging package -
> > > knative.dev/pkg/logging
> > > > - pkg.go.dev  > >).
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Feb 16, 2022 at 5:50 PM Seth Wiesman 
> > > wrote:
> > > >
> > > >> Hi Galen,
> > > >>
> > > >> No, that is not currently supported, the current idiomatic way would
> > be
> > > to
> > > >> pass those values to the struct implementing the Statefun interface.
> > > >>
> > > >>
> > > >> type MyFunc struct { someRuntimeInfo string } func (m *MyFunc)
> > > Invoke(ctx
> > > >> statefun.Context, message statefun.Message) error { } func main() {
> > > >> builder
> > > >> := statefun.StatefulFunctionsBuilder()
> > > >> f := MyFunc { someRuntimeInfo: "runtime-provided" } builder.WithSpec
> > > >> (statefun.StatefulFunctionSpec{ FunctionType: statefun.TypeNameFrom(
> > > >> "example/my-func"), Function: f })
> > > >> http.Handle("/statefun", builder.AsHandler())
> > > >> _ = http.ListenAndServe(":8000", nil) }
> > > >>
> > > >> Would this work for you? Or what is the context (pun intended) you
> are
> > > >> looking for?
> > > >>
> > > >> Seth
> > > >>
> > > >> On Wed, Feb 16, 2022 at 4:35 PM Galen Warren <
> ga...@cvillewarrens.com
> > >
> > > >> wrote:
> > > >>
> > > >> > When stateful functions are invoked, they are passed an instance
> of
> > > >> > statefun.Context, which wraps the context.Context received by the
> > HTTP
> > > >> > request. Is there any way to customize that context.Context to,
> say,
> > > >> hold
> > > >> > custom values, using ctx.WithValue()? I don't see a way but I
> wanted
> > > to
> > > >> > ask.
> > > >> >
> > > >> > If not, would you be interested in a PR to add this
> functionality? A
> > > >> simple
> > > >> > way might be to add a property to StatefulFunctionSpec, say:
> > > >> >
> > > >> > TransformContext func(ctx context.Context) context.Context
> > > >> >
> > > >> > ... that, if supplied, would be called to create a customized
> > context
> > > >> that
> > > >> > would be used downstream?
> > > >> >
> > > >> > Thanks.
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [ANNOUNCE] Apache Flink 1.13.6 released

[Till Rohrmann]

Thanks Konstantin for being our release manager and also thanks a lot to
everyone who contributed to the release!

Cheers,
Till

On Fri, Feb 18, 2022 at 2:54 PM Johannes Moser  wrote:

> Thanks. Great job.
>
> > On 18.02.2022, at 14:36, Martijn Visser  wrote:
> >
> > Hi Konstantin,
> >
> > Thanks for driving the release. Thanks to everyone that has contributed
> > with bug fixes and other improvements!
> >
> > Best regards,
> >
> > Martijn Visser
> > https://twitter.com/MartijnVisser82
> >
> >
> > On Fri, 18 Feb 2022 at 14:32, Konstantin Knauf 
> wrote:
> >
> >> The Apache Flink community is very happy to announce the release of
> Apache
> >> Flink 1.13.6, which is the fifth bugfix release for the Apache Flink
> 1.13
> >> series.
> >>
> >> Apache Flink® is an open-source stream processing framework for
> >> distributed, high-performing, always-available, and accurate data
> streaming
> >> applications.
> >>
> >> The release is available for download at:
> >> https://flink.apache.org/downloads.html
> >>
> >> Please check out the release blog post for an overview of the
> improvements
> >> for this bugfix release:
> >>
> >> https://flink.apache.org/news/2022/02/09/release-1.13.6.html
> >>
> >> The full release notes are available in Jira:
> >>
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351074
> >>
> >> We would like to thank all contributors of the Apache Flink community
> who
> >> made this release possible!
> >>
> >> Regards,
> >>
> >> Konstantin
> >>
> >> --
> >>
> >> Konstantin Knauf
> >>
> >> https://twitter.com/snntrable
> >>
> >> https://github.com/knaufk
> >>
>
>

Re: [DISCUSS] Plan to externalize connectors and versioning

[Chesnay Schepler]

Why do you want to immediately do a release for the elasticsearch 
connector? What does that provide us?


I'd rather first have a fully working setup and integrated documentation 
before thinking about releasing anything.
Once we have that we may be able to externalize all connectors within 1 
release cycle and do a clean switch; otherwise we end up with a bit of a 
messy situation for users where some connectors use version scheme A and 
others B.


I also doubt the value of 6). They'll have to update the version anyway 
(and discover at some point that the version scheme has changed), so I 
don't see what this makes easier.


On 18/02/2022 14:54, Martijn Visser wrote:

Hi everyone,

As a follow-up to earlier discussions [1] [2] to externalize the connectors
from the Flink repository, I would like to propose a plan to externalize
these connectors. The goal of this plan is to start with moving connectors
to its own repositories without introducing regressions for connector
developers.

The plan is as follows:

1. A new repository is requested for a connector.
2. The code for that connector is moved to its individual repository,
including the commit history
3. Any first release made for a connector in an external connector
repository starts with major version 3, so 3.0.0. The reason for that is
that we want to decouple the Flink releases from a connector release. If we
would start with major version 2, it could cause some confusion because
people could think a Flink 2.0 has been released. This does mean that each
connector needs to have a compatibility matrix generated, stating which
version number of the connector is compatible with the correct Flink
version.
4. The group id and artifact id for the connector will remain the same,
only the version is different.
5. The connector dependencies on the Flink website are updated to point to
the newly released connector artifact.
6. If a connector is moved, there is one release cycle where there will be
binary releases for that connector in both Flink core and from the
connector repository. This is to make Flink users who are upgrading
slightly easier. We will have to make a note in the release notes that a
connector has been moved and that a user should update any references from
the original connector artifact (from the Flink connector) to the new
connector artifact (from the external conenctor version)

We propose to first try to execute this plan for the Elasticsearch
connector as follows:

1. We wait until the Flink 1.15 release branch is cut
2. When that's done, the Elasticsearch code (including commit history) from
Flink's 1.15 release branch will be moved to the
flink-connector-elasticsearch main branch.
3. When Flink 1.15 is released, we will also release an Elasticsearch
connector for the external connector repository with version 3.0.0.
4. Bugfixes or improvements will be made first pointing to the external
connector repository and will be cherry-picked back to the release-1.15
branch in the Flink core repository.
5. The Elasticsearch code, test etc will be removed from the master branch
in the Flink core repository and dropped with Flink 1.16

Looking forward to your thoughts on this!

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82

[1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
[2] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr

[jira] [Created] (FLINK-26255) SplitAggregateITCase.testAggWithJoin failed on azure

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-26255:
-

 Summary: SplitAggregateITCase.testAggWithJoin failed on azure
 Key: FLINK-26255
 URL: https://issues.apache.org/jira/browse/FLINK-26255
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Runtime
Affects Versions: 1.15.0
Reporter: Roman Khachatryan
 Fix For: 1.15.0


[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=31850=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=10497]

 
{code:java}
[ERROR] Tests run: 64, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
700.545 s <<< FAILURE! - in 
org.apache.flink.table.planner.runtime.stream.sql.SplitAggregateITCase
[ERROR] SplitAggregateITCase.testAggWithJoin  Time elapsed: 601.77 s  <<< ERROR!
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
   at 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
   at 
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniCl
 usterJobClient.java:141)
   at 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
   at 
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
   at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
   at 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
   at 
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandle
 r.java:259)
   at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
   at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
   at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
   at 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
   at 
org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
   at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java
 :93)
   at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadi
 ngUtils.java:68)
   at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextCla
 ssLoader$2(ClassLoadingUtils.java:92)
   at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
   at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
   at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
   at 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
   at 
org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
   at akka.dispatch.OnComplete.internal(Future.scala:300)
   at akka.dispatch.OnComplete.internal(Future.scala:297)
   at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
   at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
   at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
   at 
org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFut
 ureUtils.java:65)
   at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
   at 
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
   at 
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
   at 
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: question about StatefunContext in golang Statefun SDK

[Galen Warren]

The former.

I think there's potential for confusion here because we're using the
word *function
*in a couple of senses. One sense is a *stateful function*; another sense
is a *Go function*.

What I'm looking to do is to put values in the Context so that downstream
Go functions that receive the context can access those values. Those
downstream Go functions would be called during one invocation of the
stateful function.

On Fri, Feb 18, 2022 at 6:48 AM Till Rohrmann  wrote:

> Hi Galen,
>
> Am I understanding it correctly, that you would like to set some values in
> the Context of function A that is then accessible in a downstream call of
> function B? Or would you like to set a value that is accessible once
> function A is called again (w/ or w/o the same id)?
>
> Cheers,
> Till
>
> On Thu, Feb 17, 2022 at 10:59 PM Galen Warren 
> wrote:
>
> > Also, a potentially simpler way to support this would be to add a
> > SetContext method to the statefun.Context interface, and have it assign
> the
> > wrapped context. This would not require changes to the function spec, or
> > anything else, and would be more flexible.
> >
> > On Thu, Feb 17, 2022 at 1:05 PM Galen Warren 
> > wrote:
> >
> > > Thanks for the quick reply!
> > >
> > > What I'm trying to do is put some things into the context so that
> they're
> > > available in downstream calls, perhaps in methods with pointer
> receivers
> > to
> > > the function struct (MyFunc) but also perhaps in methods that are
> further
> > > downstream that don't have access to MyFunc. If I'm understanding
> > > correctly, your proposal would work for the former but not the latter.
> > >
> > > An example would be to put a configured Logger into the context via a
> > > WithLogger method (logging package - knative.dev/pkg/logging -
> > pkg.go.dev
> > > ) and then pull
> > it
> > > out downstream via FromContext (logging package -
> > knative.dev/pkg/logging
> > > - pkg.go.dev  >).
> > >
> > >
> > >
> > >
> > > On Wed, Feb 16, 2022 at 5:50 PM Seth Wiesman 
> > wrote:
> > >
> > >> Hi Galen,
> > >>
> > >> No, that is not currently supported, the current idiomatic way would
> be
> > to
> > >> pass those values to the struct implementing the Statefun interface.
> > >>
> > >>
> > >> type MyFunc struct { someRuntimeInfo string } func (m *MyFunc)
> > Invoke(ctx
> > >> statefun.Context, message statefun.Message) error { } func main() {
> > >> builder
> > >> := statefun.StatefulFunctionsBuilder()
> > >> f := MyFunc { someRuntimeInfo: "runtime-provided" } builder.WithSpec
> > >> (statefun.StatefulFunctionSpec{ FunctionType: statefun.TypeNameFrom(
> > >> "example/my-func"), Function: f })
> > >> http.Handle("/statefun", builder.AsHandler())
> > >> _ = http.ListenAndServe(":8000", nil) }
> > >>
> > >> Would this work for you? Or what is the context (pun intended) you are
> > >> looking for?
> > >>
> > >> Seth
> > >>
> > >> On Wed, Feb 16, 2022 at 4:35 PM Galen Warren  >
> > >> wrote:
> > >>
> > >> > When stateful functions are invoked, they are passed an instance of
> > >> > statefun.Context, which wraps the context.Context received by the
> HTTP
> > >> > request. Is there any way to customize that context.Context to, say,
> > >> hold
> > >> > custom values, using ctx.WithValue()? I don't see a way but I wanted
> > to
> > >> > ask.
> > >> >
> > >> > If not, would you be interested in a PR to add this functionality? A
> > >> simple
> > >> > way might be to add a property to StatefulFunctionSpec, say:
> > >> >
> > >> > TransformContext func(ctx context.Context) context.Context
> > >> >
> > >> > ... that, if supplied, would be called to create a customized
> context
> > >> that
> > >> > would be used downstream?
> > >> >
> > >> > Thanks.
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] Plan to externalize connectors and versioning

[Johannes Moser]

+1 (non-binding)

> On 18.02.2022, at 15:12, Konstantin Knauf  wrote:
> 
> +1 to the approach.
> 
> I expect that we will encounter more questions and challenges as we go,
> but these are best discussed and addressed in the context of a specific
> connector like ElasticSearch.
> 
> On Fri, Feb 18, 2022 at 2:54 PM Martijn Visser 
> wrote:
> 
>> Hi everyone,
>> 
>> As a follow-up to earlier discussions [1] [2] to externalize the connectors
>> from the Flink repository, I would like to propose a plan to externalize
>> these connectors. The goal of this plan is to start with moving connectors
>> to its own repositories without introducing regressions for connector
>> developers.
>> 
>> The plan is as follows:
>> 
>> 1. A new repository is requested for a connector.
>> 2. The code for that connector is moved to its individual repository,
>> including the commit history
>> 3. Any first release made for a connector in an external connector
>> repository starts with major version 3, so 3.0.0. The reason for that is
>> that we want to decouple the Flink releases from a connector release. If we
>> would start with major version 2, it could cause some confusion because
>> people could think a Flink 2.0 has been released. This does mean that each
>> connector needs to have a compatibility matrix generated, stating which
>> version number of the connector is compatible with the correct Flink
>> version.
>> 4. The group id and artifact id for the connector will remain the same,
>> only the version is different.
>> 5. The connector dependencies on the Flink website are updated to point to
>> the newly released connector artifact.
>> 6. If a connector is moved, there is one release cycle where there will be
>> binary releases for that connector in both Flink core and from the
>> connector repository. This is to make Flink users who are upgrading
>> slightly easier. We will have to make a note in the release notes that a
>> connector has been moved and that a user should update any references from
>> the original connector artifact (from the Flink connector) to the new
>> connector artifact (from the external conenctor version)
>> 
>> We propose to first try to execute this plan for the Elasticsearch
>> connector as follows:
>> 
>> 1. We wait until the Flink 1.15 release branch is cut
>> 2. When that's done, the Elasticsearch code (including commit history) from
>> Flink's 1.15 release branch will be moved to the
>> flink-connector-elasticsearch main branch.
>> 3. When Flink 1.15 is released, we will also release an Elasticsearch
>> connector for the external connector repository with version 3.0.0.
>> 4. Bugfixes or improvements will be made first pointing to the external
>> connector repository and will be cherry-picked back to the release-1.15
>> branch in the Flink core repository.
>> 5. The Elasticsearch code, test etc will be removed from the master branch
>> in the Flink core repository and dropped with Flink 1.16
>> 
>> Looking forward to your thoughts on this!
>> 
>> Best regards,
>> 
>> Martijn Visser
>> https://twitter.com/MartijnVisser82
>> 
>> [1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
>> [2] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr
>> 
> 
> 
> -- 
> 
> Konstantin Knauf
> 
> https://twitter.com/snntrable
> 
> https://github.com/knaufk

[jira] [Created] (FLINK-26254) KafkaSink might violate order of sequence numbers and risk exactly-once processing

[Fabian Paul (Jira)]

Fabian Paul created FLINK-26254:
---

 Summary: KafkaSink might violate order of sequence numbers and 
risk exactly-once processing
 Key: FLINK-26254
 URL: https://issues.apache.org/jira/browse/FLINK-26254
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.14.3, 1.15.0
Reporter: Fabian Paul


When running the KafkaSink in exactly-once mode with a very low checkpoint 
interval users are seeing `OutOfOrderSequenceException`.

It could be caused by the fact that the connector has a pool of KafkaProducers 
and the sequence numbers are not shared/reset if a new KafkaProducer tries to 
write to a partition while the previous KafkaProducer is still occupied for 
committing.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Plan to externalize connectors and versioning

[Konstantin Knauf]

+1 to the approach.

 I expect that we will encounter more questions and challenges as we go,
but these are best discussed and addressed in the context of a specific
connector like ElasticSearch.

On Fri, Feb 18, 2022 at 2:54 PM Martijn Visser 
wrote:

> Hi everyone,
>
> As a follow-up to earlier discussions [1] [2] to externalize the connectors
> from the Flink repository, I would like to propose a plan to externalize
> these connectors. The goal of this plan is to start with moving connectors
> to its own repositories without introducing regressions for connector
> developers.
>
> The plan is as follows:
>
> 1. A new repository is requested for a connector.
> 2. The code for that connector is moved to its individual repository,
> including the commit history
> 3. Any first release made for a connector in an external connector
> repository starts with major version 3, so 3.0.0. The reason for that is
> that we want to decouple the Flink releases from a connector release. If we
> would start with major version 2, it could cause some confusion because
> people could think a Flink 2.0 has been released. This does mean that each
> connector needs to have a compatibility matrix generated, stating which
> version number of the connector is compatible with the correct Flink
> version.
> 4. The group id and artifact id for the connector will remain the same,
> only the version is different.
> 5. The connector dependencies on the Flink website are updated to point to
> the newly released connector artifact.
> 6. If a connector is moved, there is one release cycle where there will be
> binary releases for that connector in both Flink core and from the
> connector repository. This is to make Flink users who are upgrading
> slightly easier. We will have to make a note in the release notes that a
> connector has been moved and that a user should update any references from
> the original connector artifact (from the Flink connector) to the new
> connector artifact (from the external conenctor version)
>
> We propose to first try to execute this plan for the Elasticsearch
> connector as follows:
>
> 1. We wait until the Flink 1.15 release branch is cut
> 2. When that's done, the Elasticsearch code (including commit history) from
> Flink's 1.15 release branch will be moved to the
> flink-connector-elasticsearch main branch.
> 3. When Flink 1.15 is released, we will also release an Elasticsearch
> connector for the external connector repository with version 3.0.0.
> 4. Bugfixes or improvements will be made first pointing to the external
> connector repository and will be cherry-picked back to the release-1.15
> branch in the Flink core repository.
> 5. The Elasticsearch code, test etc will be removed from the master branch
> in the Flink core repository and dropped with Flink 1.16
>
> Looking forward to your thoughts on this!
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
>
> [1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
> [2] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang

[Jing Zhang]

Congratulations!

Best,
Jing Zhang

Benchao Li  于2022年2月18日周五 20:15写道：

> Congratulations!
>
> Jing Ge  于2022年2月18日周五 17:03写道：
>
> > Congrats! Very well deserved!
> >
> > Best regards,
> > Jing
> >
> > On Fri, Feb 18, 2022 at 9:16 AM Yu Li  wrote:
> >
> > > Congratulations!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Fri, 18 Feb 2022 at 14:19, godfrey he  wrote:
> > >
> > > > Congratulations!
> > > >
> > > > Best,
> > > > Godfrey
> > > >
> > > > Lincoln Lee  于2022年2月18日周五 14:05写道：
> > > > >
> > > > > Congratulations Feng and Zhipeng!
> > > > >
> > > > > Best,
> > > > > Lincoln Lee
> > > > >
> > > > >
> > > > > Yuepeng Pan  于2022年2月18日周五 12:45写道：
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Congratulations!
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Yuepeng Pan
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > 在 2022-02-18 12:41:18，"Jingsong Li"  写道：
> > > > > > >Congratulations!
> > > > > > >
> > > > > > >Best,
> > > > > > >Jingsong
> > > > > > >
> > > > > > >On Fri, Feb 18, 2022 at 9:47 AM Zhipeng Zhang <
> > > > zhangzhipe...@gmail.com>
> > > > > > wrote:
> > > > > > >>
> > > > > > >> Thank you everyone for the warm welcome!
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Zhipeng
> > > > > > >>
> > > > > > >> Jinzhong Li  于2022年2月17日周四 19:58写道：
> > > > > > >>
> > > > > > >> > Congratulations!
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Best,
> > > > > > >> >
> > > > > > >> > Jinzhong
> > > > > > >> >
> > > > > > >> > Robert Metzger  于2022年2月16日周三 21:32写道：
> > > > > > >> >
> > > > > > >> > > Hi everyone,
> > > > > > >> > >
> > > > > > >> > > On behalf of the PMC, I'm very happy to announce two new
> > Flink
> > > > > > >> > > committers: Feng Wang and Zhipeng Zhang!
> > > > > > >> > >
> > > > > > >> > > Feng is one of the most active Flink evangelists in China,
> > > with
> > > > > > plenty of
> > > > > > >> > > public talks, blog posts and other evangelization
> > activities.
> > > > The
> > > > > > PMC
> > > > > > >> > wants
> > > > > > >> > > to recognize and value these efforts by making Feng a
> > > committer!
> > > > > > >> > >
> > > > > > >> > > Zhipeng Zhang has made significant contributions to
> > flink-ml,
> > > > like
> > > > > > most
> > > > > > >> > of
> > > > > > >> > > the FLIPs for our ML efforts.
> > > > > > >> > >
> > > > > > >> > > Please join me in welcoming them as committers!
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > Best,
> > > > > > >> > > Robert
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> best,
> > > > > > >> Zhipeng
> > > > > >
> > > >
> > >
> >
>
>
> --
>
> Best,
> Benchao Li
>

Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao

[Jing Zhang]

Congratulations!

Best,
Jing Zhang

Benchao Li  于2022年2月18日周五 20:22写道：

> Congratulations!
>
> Yu Li  于2022年2月18日周五 19:34写道：
>
> > Congratulations!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Fri, 18 Feb 2022 at 14:19, godfrey he  wrote:
> >
> > > Congratulations!
> > >
> > > Best,
> > > Godfrey
> > >
> > > Lincoln Lee  于2022年2月18日周五 14:07写道：
> > > >
> > > > Congratulations!
> > > >
> > > > Best,
> > > > Lincoln Lee
> > > >
> > > >
> > > > Jingsong Li  于2022年2月18日周五 12:42写道：
> > > >
> > > > > Congratulations!
> > > > >
> > > > > Best,
> > > > > Jingsong
> > > > >
> > > > > On Thu, Feb 17, 2022 at 8:08 PM Jinzhong Li <
> > lijinzhong2...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Congratulations!
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Jinzhong
> > > > > >
> > > > > > On Wed, Feb 16, 2022 at 9:23 PM Robert Metzger <
> > rmetz...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I would like to formally announce a few new Flink PMC members
> on
> > > the
> > > > > dev@
> > > > > > > list. The PMC has not done a good job of always announcing new
> > PMC
> > > > > members
> > > > > > > (and committers) recently. I'll try to keep an eye on this in
> the
> > > > > future to
> > > > > > > improve the situation.
> > > > > > >
> > > > > > > Nevertheless, I'm very happy to announce some very active
> > community
> > > > > members
> > > > > > > as new PMC members:
> > > > > > >
> > > > > > > - Igal Shilman, added to the PMC in October 2021
> > > > > > > - Konstantin Knauf, added to the PMC in January 2022
> > > > > > > - Yun Gao, added to the PMC in February 2022
> > > > > > >
> > > > > > > Please join me in welcoming them to the Flink PMC!
> > > > > > >
> > > > > > > Best,
> > > > > > > Robert
> > > > > > >
> > > > >
> > >
> >
>
>
> --
>
> Best,
> Benchao Li
>

[jira] [Created] (FLINK-26253) Speed up detection fo dropped message in MessageSerializationTest

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-26253:


 Summary: Speed up detection fo dropped message in 
MessageSerializationTest
 Key: FLINK-26253
 URL: https://issues.apache.org/jira/browse/FLINK-26253
 Project: Flink
  Issue Type: Technical Debt
  Components: Runtime / Coordination, Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.15.0


The MessageSerializationTest unnecessarily eats 10 seconds just waiting for a 
future to time out.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[DISCUSS] Plan to externalize connectors and versioning

[Martijn Visser]

Hi everyone,

As a follow-up to earlier discussions [1] [2] to externalize the connectors
from the Flink repository, I would like to propose a plan to externalize
these connectors. The goal of this plan is to start with moving connectors
to its own repositories without introducing regressions for connector
developers.

The plan is as follows:

1. A new repository is requested for a connector.
2. The code for that connector is moved to its individual repository,
including the commit history
3. Any first release made for a connector in an external connector
repository starts with major version 3, so 3.0.0. The reason for that is
that we want to decouple the Flink releases from a connector release. If we
would start with major version 2, it could cause some confusion because
people could think a Flink 2.0 has been released. This does mean that each
connector needs to have a compatibility matrix generated, stating which
version number of the connector is compatible with the correct Flink
version.
4. The group id and artifact id for the connector will remain the same,
only the version is different.
5. The connector dependencies on the Flink website are updated to point to
the newly released connector artifact.
6. If a connector is moved, there is one release cycle where there will be
binary releases for that connector in both Flink core and from the
connector repository. This is to make Flink users who are upgrading
slightly easier. We will have to make a note in the release notes that a
connector has been moved and that a user should update any references from
the original connector artifact (from the Flink connector) to the new
connector artifact (from the external conenctor version)

We propose to first try to execute this plan for the Elasticsearch
connector as follows:

1. We wait until the Flink 1.15 release branch is cut
2. When that's done, the Elasticsearch code (including commit history) from
Flink's 1.15 release branch will be moved to the
flink-connector-elasticsearch main branch.
3. When Flink 1.15 is released, we will also release an Elasticsearch
connector for the external connector repository with version 3.0.0.
4. Bugfixes or improvements will be made first pointing to the external
connector repository and will be cherry-picked back to the release-1.15
branch in the Flink core repository.
5. The Elasticsearch code, test etc will be removed from the master branch
in the Flink core repository and dropped with Flink 1.16

Looking forward to your thoughts on this!

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82

[1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
[2] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr

Re: [ANNOUNCE] Apache Flink 1.13.6 released

[Johannes Moser]

Thanks. Great job.

> On 18.02.2022, at 14:36, Martijn Visser  wrote:
> 
> Hi Konstantin,
> 
> Thanks for driving the release. Thanks to everyone that has contributed
> with bug fixes and other improvements!
> 
> Best regards,
> 
> Martijn Visser
> https://twitter.com/MartijnVisser82
> 
> 
> On Fri, 18 Feb 2022 at 14:32, Konstantin Knauf  wrote:
> 
>> The Apache Flink community is very happy to announce the release of Apache
>> Flink 1.13.6, which is the fifth bugfix release for the Apache Flink 1.13
>> series.
>> 
>> Apache Flink® is an open-source stream processing framework for
>> distributed, high-performing, always-available, and accurate data streaming
>> applications.
>> 
>> The release is available for download at:
>> https://flink.apache.org/downloads.html
>> 
>> Please check out the release blog post for an overview of the improvements
>> for this bugfix release:
>> 
>> https://flink.apache.org/news/2022/02/09/release-1.13.6.html
>> 
>> The full release notes are available in Jira:
>> 
>> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351074
>> 
>> We would like to thank all contributors of the Apache Flink community who
>> made this release possible!
>> 
>> Regards,
>> 
>> Konstantin
>> 
>> --
>> 
>> Konstantin Knauf
>> 
>> https://twitter.com/snntrable
>> 
>> https://github.com/knaufk
>> 

[jira] [Created] (FLINK-26252) Refactor MiniClusterExtension

[Francesco Guardiani (Jira)]

Francesco Guardiani created FLINK-26252:
---

 Summary: Refactor MiniClusterExtension
 Key: FLINK-26252
 URL: https://issues.apache.org/jira/browse/FLINK-26252
 Project: Flink
  Issue Type: Bug
  Components: Test Infrastructure
Reporter: Francesco Guardiani
Assignee: Francesco Guardiani






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] Apache Flink 1.13.6 released

[Martijn Visser]

Hi Konstantin,

Thanks for driving the release. Thanks to everyone that has contributed
with bug fixes and other improvements!

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82


On Fri, 18 Feb 2022 at 14:32, Konstantin Knauf  wrote:

> The Apache Flink community is very happy to announce the release of Apache
> Flink 1.13.6, which is the fifth bugfix release for the Apache Flink 1.13
> series.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Please check out the release blog post for an overview of the improvements
> for this bugfix release:
>
> https://flink.apache.org/news/2022/02/09/release-1.13.6.html
>
> The full release notes are available in Jira:
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351074
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Regards,
>
> Konstantin
>
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk
>

[ANNOUNCE] Apache Flink 1.13.6 released

[Konstantin Knauf]

The Apache Flink community is very happy to announce the release of Apache
Flink 1.13.6, which is the fifth bugfix release for the Apache Flink 1.13
series.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

Please check out the release blog post for an overview of the improvements
for this bugfix release:

https://flink.apache.org/news/2022/02/09/release-1.13.6.html

The full release notes are available in Jira:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351074

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,

Konstantin

-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao

[Benchao Li]

Congratulations!

Yu Li  于2022年2月18日周五 19:34写道：

> Congratulations!
>
> Best Regards,
> Yu
>
>
> On Fri, 18 Feb 2022 at 14:19, godfrey he  wrote:
>
> > Congratulations!
> >
> > Best,
> > Godfrey
> >
> > Lincoln Lee  于2022年2月18日周五 14:07写道：
> > >
> > > Congratulations!
> > >
> > > Best,
> > > Lincoln Lee
> > >
> > >
> > > Jingsong Li  于2022年2月18日周五 12:42写道：
> > >
> > > > Congratulations!
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > > On Thu, Feb 17, 2022 at 8:08 PM Jinzhong Li <
> lijinzhong2...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Congratulations!
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > Jinzhong
> > > > >
> > > > > On Wed, Feb 16, 2022 at 9:23 PM Robert Metzger <
> rmetz...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I would like to formally announce a few new Flink PMC members on
> > the
> > > > dev@
> > > > > > list. The PMC has not done a good job of always announcing new
> PMC
> > > > members
> > > > > > (and committers) recently. I'll try to keep an eye on this in the
> > > > future to
> > > > > > improve the situation.
> > > > > >
> > > > > > Nevertheless, I'm very happy to announce some very active
> community
> > > > members
> > > > > > as new PMC members:
> > > > > >
> > > > > > - Igal Shilman, added to the PMC in October 2021
> > > > > > - Konstantin Knauf, added to the PMC in January 2022
> > > > > > - Yun Gao, added to the PMC in February 2022
> > > > > >
> > > > > > Please join me in welcoming them to the Flink PMC!
> > > > > >
> > > > > > Best,
> > > > > > Robert
> > > > > >
> > > >
> >
>


-- 

Best,
Benchao Li

Re: Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang

[Benchao Li]

Congratulations!

Jing Ge  于2022年2月18日周五 17:03写道：

> Congrats! Very well deserved!
>
> Best regards,
> Jing
>
> On Fri, Feb 18, 2022 at 9:16 AM Yu Li  wrote:
>
> > Congratulations!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Fri, 18 Feb 2022 at 14:19, godfrey he  wrote:
> >
> > > Congratulations!
> > >
> > > Best,
> > > Godfrey
> > >
> > > Lincoln Lee  于2022年2月18日周五 14:05写道：
> > > >
> > > > Congratulations Feng and Zhipeng!
> > > >
> > > > Best,
> > > > Lincoln Lee
> > > >
> > > >
> > > > Yuepeng Pan  于2022年2月18日周五 12:45写道：
> > > >
> > > > >
> > > > >
> > > > >
> > > > > Congratulations!
> > > > >
> > > > >
> > > > > Best,
> > > > > Yuepeng Pan
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 在 2022-02-18 12:41:18，"Jingsong Li"  写道：
> > > > > >Congratulations!
> > > > > >
> > > > > >Best,
> > > > > >Jingsong
> > > > > >
> > > > > >On Fri, Feb 18, 2022 at 9:47 AM Zhipeng Zhang <
> > > zhangzhipe...@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> Thank you everyone for the warm welcome!
> > > > > >>
> > > > > >> Best,
> > > > > >> Zhipeng
> > > > > >>
> > > > > >> Jinzhong Li  于2022年2月17日周四 19:58写道：
> > > > > >>
> > > > > >> > Congratulations!
> > > > > >> >
> > > > > >> >
> > > > > >> > Best,
> > > > > >> >
> > > > > >> > Jinzhong
> > > > > >> >
> > > > > >> > Robert Metzger  于2022年2月16日周三 21:32写道：
> > > > > >> >
> > > > > >> > > Hi everyone,
> > > > > >> > >
> > > > > >> > > On behalf of the PMC, I'm very happy to announce two new
> Flink
> > > > > >> > > committers: Feng Wang and Zhipeng Zhang!
> > > > > >> > >
> > > > > >> > > Feng is one of the most active Flink evangelists in China,
> > with
> > > > > plenty of
> > > > > >> > > public talks, blog posts and other evangelization
> activities.
> > > The
> > > > > PMC
> > > > > >> > wants
> > > > > >> > > to recognize and value these efforts by making Feng a
> > committer!
> > > > > >> > >
> > > > > >> > > Zhipeng Zhang has made significant contributions to
> flink-ml,
> > > like
> > > > > most
> > > > > >> > of
> > > > > >> > > the FLIPs for our ML efforts.
> > > > > >> > >
> > > > > >> > > Please join me in welcoming them as committers!
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Best,
> > > > > >> > > Robert
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> best,
> > > > > >> Zhipeng
> > > > >
> > >
> >
>


-- 

Best,
Benchao Li

Re: question about StatefunContext in golang Statefun SDK

[Till Rohrmann]

Hi Galen,

Am I understanding it correctly, that you would like to set some values in
the Context of function A that is then accessible in a downstream call of
function B? Or would you like to set a value that is accessible once
function A is called again (w/ or w/o the same id)?

Cheers,
Till

On Thu, Feb 17, 2022 at 10:59 PM Galen Warren 
wrote:

> Also, a potentially simpler way to support this would be to add a
> SetContext method to the statefun.Context interface, and have it assign the
> wrapped context. This would not require changes to the function spec, or
> anything else, and would be more flexible.
>
> On Thu, Feb 17, 2022 at 1:05 PM Galen Warren 
> wrote:
>
> > Thanks for the quick reply!
> >
> > What I'm trying to do is put some things into the context so that they're
> > available in downstream calls, perhaps in methods with pointer receivers
> to
> > the function struct (MyFunc) but also perhaps in methods that are further
> > downstream that don't have access to MyFunc. If I'm understanding
> > correctly, your proposal would work for the former but not the latter.
> >
> > An example would be to put a configured Logger into the context via a
> > WithLogger method (logging package - knative.dev/pkg/logging -
> pkg.go.dev
> > ) and then pull
> it
> > out downstream via FromContext (logging package -
> knative.dev/pkg/logging
> > - pkg.go.dev ).
> >
> >
> >
> >
> > On Wed, Feb 16, 2022 at 5:50 PM Seth Wiesman 
> wrote:
> >
> >> Hi Galen,
> >>
> >> No, that is not currently supported, the current idiomatic way would be
> to
> >> pass those values to the struct implementing the Statefun interface.
> >>
> >>
> >> type MyFunc struct { someRuntimeInfo string } func (m *MyFunc)
> Invoke(ctx
> >> statefun.Context, message statefun.Message) error { } func main() {
> >> builder
> >> := statefun.StatefulFunctionsBuilder()
> >> f := MyFunc { someRuntimeInfo: "runtime-provided" } builder.WithSpec
> >> (statefun.StatefulFunctionSpec{ FunctionType: statefun.TypeNameFrom(
> >> "example/my-func"), Function: f })
> >> http.Handle("/statefun", builder.AsHandler())
> >> _ = http.ListenAndServe(":8000", nil) }
> >>
> >> Would this work for you? Or what is the context (pun intended) you are
> >> looking for?
> >>
> >> Seth
> >>
> >> On Wed, Feb 16, 2022 at 4:35 PM Galen Warren 
> >> wrote:
> >>
> >> > When stateful functions are invoked, they are passed an instance of
> >> > statefun.Context, which wraps the context.Context received by the HTTP
> >> > request. Is there any way to customize that context.Context to, say,
> >> hold
> >> > custom values, using ctx.WithValue()? I don't see a way but I wanted
> to
> >> > ask.
> >> >
> >> > If not, would you be interested in a PR to add this functionality? A
> >> simple
> >> > way might be to add a property to StatefulFunctionSpec, say:
> >> >
> >> > TransformContext func(ctx context.Context) context.Context
> >> >
> >> > ... that, if supplied, would be called to create a customized context
> >> that
> >> > would be used downstream?
> >> >
> >> > Thanks.
> >> >
> >>
> >
>

[jira] [Created] (FLINK-26251) [JUnit5 Migration] Module: flink-rpc-akka

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-26251:


 Summary: [JUnit5 Migration] Module: flink-rpc-akka
 Key: FLINK-26251
 URL: https://issues.apache.org/jira/browse/FLINK-26251
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination, Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.15.0






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] Flink 1.15 Feature Freeze

[Till Rohrmann]

Thanks for letting us know Wenlong. Since the SQL client testing hasn't
really started yet, I think it is ok to merge this PR.

Cheers,
Till

On Thu, Feb 17, 2022 at 2:32 PM wenlong.lwl  wrote:

> Hi,all, I want to merge a pr (https://github.com/apache/flink/pull/18363)
> belonging to FLIP-190, which was approved yesterday, but not merged before
> code freeze because of CI queueing and failed by other changes.
> WDYT?
>
> Best,
> Wenlong
>
> On Thu, 17 Feb 2022 at 02:08, Till Rohrmann  wrote:
>
> > Hi everyone,
> >
> > The deadline for merging new features for Flink 1.15 has passed.
> >
> > * From now on, only bug-fixes and documentation fixes / improvements are
> > allowed to be merged into the master branch.
> >
> > * New features merged after this point can be reverted. If you need an
> > exception to this rule, please open a discussion on dev@ list and reach
> > out
> > to us.
> >
> > We plan to wait for the master branch to get a bit more stabilized before
> > cutting the "release-1.15" branch, in order to reduce the overhead of
> > having to manage two branches. That also means potentially delaying
> merging
> > new features for Flink 1.16 into the master branch. If you are blocked on
> > this, please let us know and we can come up with a compromise for the
> > branch cutting time.
> >
> > What you can do to help with the release testing phase:
> >
> > * The first release testing sync will be on *February 22, 9am CET*.
> > Everyone is welcome to join. The link can be found on the release wiki
> page
> > [1].
> >
> > * Please prepare for the release testing by creating Jira tickets for
> > documentation and testing tasks for the new features. Tickets should be
> > opened with Priority Blocker, FixVersion 1.15.0 and Label release-testing
> > (testing tasks only).
> >
> > * There are currently 92 test-stability issues affecting the 1.15.0
> release
> > [2]. It is greatly appreciated if you can help address some of them.
> >
> > Cheers,
> > Joe, Yun & Till
> >
> > [1] https://cwiki.apache.org/confluence/display/FLINK/1.15+Release
> > [2] https://issues.apache.org/jira/issues/?filter=12351363
> >
>

Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao

[Yu Li]

Congratulations!

Best Regards,
Yu


On Fri, 18 Feb 2022 at 14:19, godfrey he  wrote:

> Congratulations!
>
> Best,
> Godfrey
>
> Lincoln Lee  于2022年2月18日周五 14:07写道：
> >
> > Congratulations!
> >
> > Best,
> > Lincoln Lee
> >
> >
> > Jingsong Li  于2022年2月18日周五 12:42写道：
> >
> > > Congratulations!
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Thu, Feb 17, 2022 at 8:08 PM Jinzhong Li 
> > > wrote:
> > > >
> > > > Congratulations!
> > > >
> > > >
> > > > Best,
> > > >
> > > > Jinzhong
> > > >
> > > > On Wed, Feb 16, 2022 at 9:23 PM Robert Metzger 
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would like to formally announce a few new Flink PMC members on
> the
> > > dev@
> > > > > list. The PMC has not done a good job of always announcing new PMC
> > > members
> > > > > (and committers) recently. I'll try to keep an eye on this in the
> > > future to
> > > > > improve the situation.
> > > > >
> > > > > Nevertheless, I'm very happy to announce some very active community
> > > members
> > > > > as new PMC members:
> > > > >
> > > > > - Igal Shilman, added to the PMC in October 2021
> > > > > - Konstantin Knauf, added to the PMC in January 2022
> > > > > - Yun Gao, added to the PMC in February 2022
> > > > >
> > > > > Please join me in welcoming them to the Flink PMC!
> > > > >
> > > > > Best,
> > > > > Robert
> > > > >
> > >
>

[jira] [Created] (FLINK-26250) Document the upgrade story

[Timo Walther (Jira)]

Timo Walther created FLINK-26250:


 Summary: Document the upgrade story
 Key: FLINK-26250
 URL: https://issues.apache.org/jira/browse/FLINK-26250
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Timo Walther
Assignee: Timo Walther


Document FLIP-190.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26249) Run BuiltInFunctionITCase tests in parallel

[Francesco Guardiani (Jira)]

Francesco Guardiani created FLINK-26249:
---

 Summary: Run BuiltInFunctionITCase tests in parallel
 Key: FLINK-26249
 URL: https://issues.apache.org/jira/browse/FLINK-26249
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner, Test Infrastructure
Reporter: Francesco Guardiani
Assignee: Francesco Guardiani






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [NOTICE] Pulsar tests disabled

[Leonard Xu]

The Pulsar tests are too weak now and lead to Flink CI instabilities.
There’re 10+ bug issues[1] for Pulsar connector  with various error causes.


[1] https://issues.apache.org/jira/issues/?filter=12351366 
 


> 2022年2月18日 下午4:34，Yufei Zhang  写道：
> 
> Hi,
> 
> Sorry for any inconvenience and late reply,  after some investigation, the
> test instabilities are believed to be caused by the newly introduced
> PulsarEmbedRuntime, and an alternative to it is to use the
> PulsarContainerRuntime
> which runs in testcontainer.  A pr is being submitted to resolve the issue
> by using PulsarContainerRuntime, and subsequent debugging is ongoing.
> Thanks for your update, and we sincerely apologize for the situation.
> 
> 
> 
> Cheers.
> 
> On Fri, Feb 18, 2022 at 4:06 PM Chesnay Schepler  wrote:
> 
>> Given these instabilities we might also want to consider excluding it
>> from the 1.15 release, because it is clearly not in a reasonable stable
>> state before the feature freeze.
>> 
>> On 18/02/2022 08:58, Chesnay Schepler wrote:
>>> FYI: I just disabled all Pulsar tests on master in
>>> 296c3be7576940d6bd76f4d065a0b70bceb06991.
>>> 
>>> They are currently the main source of test instabilities, and I don't
>>> see any real efforts or progress on those being addressed.
>>> 
>> 
>> 

Re: [NOTICE] Pulsar tests disabled

[Yufei Zhang]

Hi,

Here are some updates regarding the ongoing pulsar test instabilities.

The following issues are believed to be caused by PulsarEmbeddedRuntime or
mockRuntime, and currently, we have a PR

https://github.com/apache/flink/pull/18838/commits to use
PulsarContainerRuntime. Suspected causes are

   - FLINK-25815: mock pulsar runtime, changed to test containers.
   - FLINK-26177
   - FLINK-26192
   - FLINK-26237
   - FLINK-26240
   - FLINK-26238

The following two tickets are tracked by the corresponding PR

   - FLINK-26241: Divide zero, a bug in test code. Tracked by
   
   https://github.com/apache/flink/pull/18836
   - FLINK-26210: Pulsar e2e tests with jaxb-api. Tracked by
   
   https://github.com/apache/flink/pull/18815

Cheers.

On Fri, Feb 18, 2022 at 4:34 PM Yufei Zhang  wrote:

> Hi,
>
> Sorry for any inconvenience and late reply,  after some investigation, the
> test instabilities are believed to be caused by the newly introduced
> PulsarEmbedRuntime, and an alternative to it is to use the
> PulsarContainerRuntime
> which runs in testcontainer.  A pr is being submitted to resolve the issue
> by using PulsarContainerRuntime, and subsequent debugging is ongoing.
> Thanks for your update, and we sincerely apologize for the situation.
>
>
>
> Cheers.
>
> On Fri, Feb 18, 2022 at 4:06 PM Chesnay Schepler 
> wrote:
>
>> Given these instabilities we might also want to consider excluding it
>> from the 1.15 release, because it is clearly not in a reasonable stable
>> state before the feature freeze.
>>
>> On 18/02/2022 08:58, Chesnay Schepler wrote:
>> > FYI: I just disabled all Pulsar tests on master in
>> > 296c3be7576940d6bd76f4d065a0b70bceb06991.
>> >
>> > They are currently the main source of test instabilities, and I don't
>> > see any real efforts or progress on those being addressed.
>> >
>>
>>

[jira] [Created] (FLINK-26248) flink job not recover from latest checkpoint on native k8s

[zhisheng (Jira)]

zhisheng created FLINK-26248:


 Summary: flink job not recover from latest checkpoint on native k8s
 Key: FLINK-26248
 URL: https://issues.apache.org/jira/browse/FLINK-26248
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes
Affects Versions: 1.12.0
Reporter: zhisheng


flink version: 1.12.0

ha: zk

mode: application mode

native k8s

 

flink job start with savepoint, running for a while，then i delete the JM pod, 
the job will restart with the old savepoint not the latest checkpoint.  this is 
not what i want

if i delete the TM pod, the job will restart with the  latest the 
checkpoint，this is what i want



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang

[Jing Ge]

Congrats! Very well deserved!

Best regards,
Jing

On Fri, Feb 18, 2022 at 9:16 AM Yu Li  wrote:

> Congratulations!
>
> Best Regards,
> Yu
>
>
> On Fri, 18 Feb 2022 at 14:19, godfrey he  wrote:
>
> > Congratulations!
> >
> > Best,
> > Godfrey
> >
> > Lincoln Lee  于2022年2月18日周五 14:05写道：
> > >
> > > Congratulations Feng and Zhipeng!
> > >
> > > Best,
> > > Lincoln Lee
> > >
> > >
> > > Yuepeng Pan  于2022年2月18日周五 12:45写道：
> > >
> > > >
> > > >
> > > >
> > > > Congratulations!
> > > >
> > > >
> > > > Best,
> > > > Yuepeng Pan
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 在 2022-02-18 12:41:18，"Jingsong Li"  写道：
> > > > >Congratulations!
> > > > >
> > > > >Best,
> > > > >Jingsong
> > > > >
> > > > >On Fri, Feb 18, 2022 at 9:47 AM Zhipeng Zhang <
> > zhangzhipe...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> Thank you everyone for the warm welcome!
> > > > >>
> > > > >> Best,
> > > > >> Zhipeng
> > > > >>
> > > > >> Jinzhong Li  于2022年2月17日周四 19:58写道：
> > > > >>
> > > > >> > Congratulations!
> > > > >> >
> > > > >> >
> > > > >> > Best,
> > > > >> >
> > > > >> > Jinzhong
> > > > >> >
> > > > >> > Robert Metzger  于2022年2月16日周三 21:32写道：
> > > > >> >
> > > > >> > > Hi everyone,
> > > > >> > >
> > > > >> > > On behalf of the PMC, I'm very happy to announce two new Flink
> > > > >> > > committers: Feng Wang and Zhipeng Zhang!
> > > > >> > >
> > > > >> > > Feng is one of the most active Flink evangelists in China,
> with
> > > > plenty of
> > > > >> > > public talks, blog posts and other evangelization activities.
> > The
> > > > PMC
> > > > >> > wants
> > > > >> > > to recognize and value these efforts by making Feng a
> committer!
> > > > >> > >
> > > > >> > > Zhipeng Zhang has made significant contributions to flink-ml,
> > like
> > > > most
> > > > >> > of
> > > > >> > > the FLIPs for our ML efforts.
> > > > >> > >
> > > > >> > > Please join me in welcoming them as committers!
> > > > >> > >
> > > > >> > >
> > > > >> > > Best,
> > > > >> > > Robert
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >> --
> > > > >> best,
> > > > >> Zhipeng
> > > >
> >
>

[jira] [Created] (FLINK-26247) Introduce a better expire strategy

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-26247:


 Summary: Introduce a better expire strategy
 Key: FLINK-26247
 URL: https://issues.apache.org/jira/browse/FLINK-26247
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.1.0


We can add the snapshot id in which the file/manifest was added to the table. 
With this snapshot id, we can have better expire strategy. Instead of scanning 
all files of the snapshot.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26246) Setup JUnit 5 infrastructure for parallel test execution

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-26246:


 Summary: Setup JUnit 5 infrastructure for parallel test execution
 Key: FLINK-26246
 URL: https://issues.apache.org/jira/browse/FLINK-26246
 Project: Flink
  Issue Type: Technical Debt
  Components: Build System, Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.15.0


JUnit5 has a nifty feature that allows tests to be run in parallel. We can set 
this up such that any JUnit5 test can easily opt-in to that, which will allow 
us to significantly speed up CI.

This is currently only set up for flink-table-planner, but we can pull it into 
the root pom.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [NOTICE] Pulsar tests disabled

[Yufei Zhang]

Hi,

Sorry for any inconvenience and late reply,  after some investigation, the
test instabilities are believed to be caused by the newly introduced
PulsarEmbedRuntime, and an alternative to it is to use the
PulsarContainerRuntime
which runs in testcontainer.  A pr is being submitted to resolve the issue
by using PulsarContainerRuntime, and subsequent debugging is ongoing.
Thanks for your update, and we sincerely apologize for the situation.



Cheers.

On Fri, Feb 18, 2022 at 4:06 PM Chesnay Schepler  wrote:

> Given these instabilities we might also want to consider excluding it
> from the 1.15 release, because it is clearly not in a reasonable stable
> state before the feature freeze.
>
> On 18/02/2022 08:58, Chesnay Schepler wrote:
> > FYI: I just disabled all Pulsar tests on master in
> > 296c3be7576940d6bd76f4d065a0b70bceb06991.
> >
> > They are currently the main source of test instabilities, and I don't
> > see any real efforts or progress on those being addressed.
> >
>
>

Re: Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang

[Yu Li]

Congratulations!

Best Regards,
Yu


On Fri, 18 Feb 2022 at 14:19, godfrey he  wrote:

> Congratulations!
>
> Best,
> Godfrey
>
> Lincoln Lee  于2022年2月18日周五 14:05写道：
> >
> > Congratulations Feng and Zhipeng!
> >
> > Best,
> > Lincoln Lee
> >
> >
> > Yuepeng Pan  于2022年2月18日周五 12:45写道：
> >
> > >
> > >
> > >
> > > Congratulations!
> > >
> > >
> > > Best,
> > > Yuepeng Pan
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 在 2022-02-18 12:41:18，"Jingsong Li"  写道：
> > > >Congratulations!
> > > >
> > > >Best,
> > > >Jingsong
> > > >
> > > >On Fri, Feb 18, 2022 at 9:47 AM Zhipeng Zhang <
> zhangzhipe...@gmail.com>
> > > wrote:
> > > >>
> > > >> Thank you everyone for the warm welcome!
> > > >>
> > > >> Best,
> > > >> Zhipeng
> > > >>
> > > >> Jinzhong Li  于2022年2月17日周四 19:58写道：
> > > >>
> > > >> > Congratulations!
> > > >> >
> > > >> >
> > > >> > Best,
> > > >> >
> > > >> > Jinzhong
> > > >> >
> > > >> > Robert Metzger  于2022年2月16日周三 21:32写道：
> > > >> >
> > > >> > > Hi everyone,
> > > >> > >
> > > >> > > On behalf of the PMC, I'm very happy to announce two new Flink
> > > >> > > committers: Feng Wang and Zhipeng Zhang!
> > > >> > >
> > > >> > > Feng is one of the most active Flink evangelists in China, with
> > > plenty of
> > > >> > > public talks, blog posts and other evangelization activities.
> The
> > > PMC
> > > >> > wants
> > > >> > > to recognize and value these efforts by making Feng a committer!
> > > >> > >
> > > >> > > Zhipeng Zhang has made significant contributions to flink-ml,
> like
> > > most
> > > >> > of
> > > >> > > the FLIPs for our ML efforts.
> > > >> > >
> > > >> > > Please join me in welcoming them as committers!
> > > >> > >
> > > >> > >
> > > >> > > Best,
> > > >> > > Robert
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> best,
> > > >> Zhipeng
> > >
>

Re: [NOTICE] Pulsar tests disabled

[Chesnay Schepler]

Given these instabilities we might also want to consider excluding it 
from the 1.15 release, because it is clearly not in a reasonable stable 
state before the feature freeze.


On 18/02/2022 08:58, Chesnay Schepler wrote:
FYI: I just disabled all Pulsar tests on master in 
296c3be7576940d6bd76f4d065a0b70bceb06991.


They are currently the main source of test instabilities, and I don't 
see any real efforts or progress on those being addressed.