[FLINK-14076] non-deserializable root cause in DeclineCheckpoint

[Jeffrey Martin]

(possible dupe; I wasn't subscribed before and the previous message didn't
seem to go through)

I'm on Flink v1.9 with the Kafka connector and a standalone JM.
If FlinkKafkaProducer fails while checkpointing, it throws a KafkaException
which gets wrapped in a CheckpointException which is sent to the JM as a
DeclineCheckpoint. KafkaException isn't on the JM default classpath, so the
JM throws a fairly cryptic ClassNotFoundException. The details of the
KafkaException wind up suppressed so it's impossible to figure out what
actually went wrong.

I can think of two fixes that would prevent this from occurring in the
Kafka or other connectors in the future:
1. DeclineCheckpoint should always send a SerializedThrowable to the JM
rather than allowing CheckpointExceptions with non-deserializable root
causes to slip through
2. CheckpointException should always capture its wrapped exception as a
SerializedThrowable (i.e., use 'super(new SerializedThrowable(cause))'
rather than 'super(cause)').

Thoughts?

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

[Xuefu Z]

+1 for using the keyword "SYSTEM". Thanks to Timo for chiming in!

--Xuefu

On Fri, Sep 20, 2019 at 3:28 PM Timo Walther  wrote:

> Hi everyone,
>
> sorry, for the late replay. I give also +1 for option #2. Thus, I guess
> we have a clear winner.
>
> I would also like to find a better keyword/syntax for this statement.
> Esp. the BUILTIN keyword can confuse people, because it could be written
> as BUILTIN, BUILDIN, BUILT_IN, or BUILD_IN. And we would need to
> introduce a new reserved keyword in the parser which affects also
> non-DDL queries. How about:
>
> CREATE TEMPORARY SYSTEM FUNCTION xxx
>
> The SYSTEM keyword is already a reserved keyword and in FLIP-66 we are
> discussing to prefix some of the function with a SYSTEM_ prefix like
> SYSTEM_WATERMARK. Also SQL defines syntax like "FOR SYSTEM_TIME AS OF".
>
> What do you think?
>
> Thanks,
> Timo
>
>
> On 20.09.19 05:45, Bowen Li wrote:
> > Another reason I prefer "CREATE TEMPORARY BUILTIN FUNCTION" over "ALTER
> > BUILTIN FUNCTION xxx TEMPORARILY" is - what if users want to drop the
> > temporary built-in function in the same session? With the former one,
> they
> > can run something like "DROP TEMPORARY BUILTIN FUNCTION"; With the latter
> > one, I'm not sure how users can "restore" the original builtin function
> > easily from an "altered" function without introducing further nonstandard
> > SQL syntax.
> >
> > Also please pardon me as I realized using net may not be a good idea...
> I'm
> > trying to fit this vote into cases listed in Flink Bylaw [1].
> >
> > >From the following result, the majority seems to be #2 too as it has the
> > most approval so far and doesn't have strong "-1".
> >
> > #1：3 (+1), 1 (0), 4(-1)
> > #2：4(0), 3 (+1), 1(+0.5)
> > * Dawid -1/0 depending on keyword
> > #3：2(+1), 3(-1), 3(0)
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
> >
> > On Thu, Sep 19, 2019 at 10:30 AM Bowen Li  wrote:
> >
> >> Hi,
> >>
> >> Thanks everyone for your votes. I summarized the result as following:
> >>
> >> #1：3 (+1), 1 (0), 4(-1) - net: -1
> >> #2：4(0), 2 (+1), 1(+0.5)  - net: +2.5
> >>  Dawid -1/0 depending on keyword
> >> #3：2(+1), 3(-1), 3(0)   - net: -1
> >>
> >> Given the result, I'd like to change my vote for #2 from 0 to +1, to
> make
> >> it a stronger case with net +3.5. So the votes so far are:
> >>
> >> #1：3 (+1), 1 (0), 4(-1) - net: -1
> >> #2：4(0), 3 (+1), 1(+0.5)  - net: +3.5
> >>  Dawid -1/0 depending on keyword
> >> #3：2(+1), 3(-1), 3(0)   - net: -1
> >>
> >> What do you think? Do you think we can conclude with this result? Or
> would
> >> you like to take it as a formal FLIP vote with 3 days voting period?
> >>
> >> BTW, I'd prefer "CREATE TEMPORARY BUILTIN FUNCTION" over "ALTER BUILTIN
> >> FUNCTION xxx TEMPORARILY" because
> >> 1. the syntax is more consistent with "CREATE FUNCTION" and "CREATE
> >> TEMPORARY FUNCTION"
> >> 2. "ALTER BUILTIN FUNCTION xxx TEMPORARILY" implies it alters a built-in
> >> function but it actually doesn't, the logic only creates a temp function
> >> with higher priority than that built-in function in ambiguous resolution
> >> order; and it would behave inconsistently with "ALTER FUNCTION".
> >>
> >>
> >>
> >> On Thu, Sep 19, 2019 at 2:58 AM Fabian Hueske 
> wrote:
> >>
> >>> I agree, it's very similar from the implementation point of view and
> the
> >>> implications.
> >>>
> >>> IMO, the difference is mostly on the mental model for the user.
> >>> Instead of having a special class of temporary functions that have
> >>> precedence over builtin functions it suggests to temporarily change
> >>> built-in functions.
> >>>
> >>> Fabian
> >>>
> >>> Am Do., 19. Sept. 2019 um 11:52 Uhr schrieb Kurt Young <
> ykt...@gmail.com
>  :
>  Hi Fabian,
> 
>  I think it's almost the same with #2 with different keyword:
> 
>  CREATE TEMPORARY BUILTIN FUNCTION xxx
> 
>  Best,
>  Kurt
> 
> 
>  On Thu, Sep 19, 2019 at 5:50 PM Fabian Hueske 
> >>> wrote:
> > Hi,
> >
> > I thought about it a bit more and think that there is some good value
> >>> in
>  my
> > last proposal.
> >
> > A lot of complexity comes from the fact that we want to allow
> >>> overriding
> > built-in functions which are differently addressed as other functions
>  (and
> > db objects).
> > We could just have "CREATE TEMPORARY FUNCTION" do exactly the same
> >>> thing
>  as
> > "CREATE FUNCTION" and treat both functions exactly the same except
> >>> that:
> > 1) temp functions disappear at the end of the session
> > 2) temp function are resolved before other functions
> >
> > This would be Dawid's proposal from the beginning of this thread (in
> >>> case
> > you still remember... ;-) )
> >
> > Temporarily overriding built-in functions would be supported with an
> > explicit command like
> >
> > ALTER BUILTIN FUNCTION xxx 

[jira] [Created] (FLINK-14158) Update Mesos configs to add leaseOfferExpiration and declinedOfferRefuse durations

[Piyush Narang (Jira)]

Piyush Narang created FLINK-14158:
-

 Summary: Update Mesos configs to add leaseOfferExpiration and 
declinedOfferRefuse durations
 Key: FLINK-14158
 URL: https://issues.apache.org/jira/browse/FLINK-14158
 Project: Flink
  Issue Type: Bug
Reporter: Piyush Narang


While debugging some Flink on Mesos scheduling issues (tied to our use of Mesos 
quotas) we end up getting skewed offers that are useless fairly often. As we 
are not rejecting these offers fast enough and as we are not telling Mesos to 
not re-send for a long enough period, we end up not being able to schedule our 
job for upwards of an hour (~30 Mesos containers). 

The Fenzo default is to reject expired and unused Mesos offers after 120s, this 
can be overridden using their TaskScheduler builder. Additionally, Mesos allows 
us to override the time for which it won't re-send offers (default is 5s). We 
found that updating to reject more aggressively (every 1s instead of 120s) and 
keeping rejected offers away for longer (60s instead of 5s) dramatically 
increases our chances of scheduling our jobs on Mesos. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

[Timo Walther]

Hi everyone,

sorry, for the late replay. I give also +1 for option #2. Thus, I guess 
we have a clear winner.


I would also like to find a better keyword/syntax for this statement. 
Esp. the BUILTIN keyword can confuse people, because it could be written 
as BUILTIN, BUILDIN, BUILT_IN, or BUILD_IN. And we would need to 
introduce a new reserved keyword in the parser which affects also 
non-DDL queries. How about:


CREATE TEMPORARY SYSTEM FUNCTION xxx

The SYSTEM keyword is already a reserved keyword and in FLIP-66 we are 
discussing to prefix some of the function with a SYSTEM_ prefix like 
SYSTEM_WATERMARK. Also SQL defines syntax like "FOR SYSTEM_TIME AS OF".


What do you think?

Thanks,
Timo


On 20.09.19 05:45, Bowen Li wrote:

Another reason I prefer "CREATE TEMPORARY BUILTIN FUNCTION" over "ALTER
BUILTIN FUNCTION xxx TEMPORARILY" is - what if users want to drop the
temporary built-in function in the same session? With the former one, they
can run something like "DROP TEMPORARY BUILTIN FUNCTION"; With the latter
one, I'm not sure how users can "restore" the original builtin function
easily from an "altered" function without introducing further nonstandard
SQL syntax.

Also please pardon me as I realized using net may not be a good idea... I'm
trying to fit this vote into cases listed in Flink Bylaw [1].

>From the following result, the majority seems to be #2 too as it has the
most approval so far and doesn't have strong "-1".

#1：3 (+1), 1 (0), 4(-1)
#2：4(0), 3 (+1), 1(+0.5)
* Dawid -1/0 depending on keyword
#3：2(+1), 3(-1), 3(0)

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026

On Thu, Sep 19, 2019 at 10:30 AM Bowen Li  wrote:


Hi,

Thanks everyone for your votes. I summarized the result as following:

#1：3 (+1), 1 (0), 4(-1) - net: -1
#2：4(0), 2 (+1), 1(+0.5)  - net: +2.5
 Dawid -1/0 depending on keyword
#3：2(+1), 3(-1), 3(0)   - net: -1

Given the result, I'd like to change my vote for #2 from 0 to +1, to make
it a stronger case with net +3.5. So the votes so far are:

#1：3 (+1), 1 (0), 4(-1) - net: -1
#2：4(0), 3 (+1), 1(+0.5)  - net: +3.5
 Dawid -1/0 depending on keyword
#3：2(+1), 3(-1), 3(0)   - net: -1

What do you think? Do you think we can conclude with this result? Or would
you like to take it as a formal FLIP vote with 3 days voting period?

BTW, I'd prefer "CREATE TEMPORARY BUILTIN FUNCTION" over "ALTER BUILTIN
FUNCTION xxx TEMPORARILY" because
1. the syntax is more consistent with "CREATE FUNCTION" and "CREATE
TEMPORARY FUNCTION"
2. "ALTER BUILTIN FUNCTION xxx TEMPORARILY" implies it alters a built-in
function but it actually doesn't, the logic only creates a temp function
with higher priority than that built-in function in ambiguous resolution
order; and it would behave inconsistently with "ALTER FUNCTION".



On Thu, Sep 19, 2019 at 2:58 AM Fabian Hueske  wrote:


I agree, it's very similar from the implementation point of view and the
implications.

IMO, the difference is mostly on the mental model for the user.
Instead of having a special class of temporary functions that have
precedence over builtin functions it suggests to temporarily change
built-in functions.

Fabian

Am Do., 19. Sept. 2019 um 11:52 Uhr schrieb Kurt Young 
:
Hi Fabian,

I think it's almost the same with #2 with different keyword:

CREATE TEMPORARY BUILTIN FUNCTION xxx

Best,
Kurt


On Thu, Sep 19, 2019 at 5:50 PM Fabian Hueske 

wrote:

Hi,

I thought about it a bit more and think that there is some good value

in

my

last proposal.

A lot of complexity comes from the fact that we want to allow

overriding

built-in functions which are differently addressed as other functions

(and

db objects).
We could just have "CREATE TEMPORARY FUNCTION" do exactly the same

thing

as

"CREATE FUNCTION" and treat both functions exactly the same except

that:

1) temp functions disappear at the end of the session
2) temp function are resolved before other functions

This would be Dawid's proposal from the beginning of this thread (in

case

you still remember... ;-) )

Temporarily overriding built-in functions would be supported with an
explicit command like

ALTER BUILTIN FUNCTION xxx TEMPORARILY AS ...

This would also address the concerns about accidentally changing the
semantics of built-in functions.
IMO, it can't get much more explicit than the above command.

Sorry for bringing up a new option in the middle of the discussion,

but

as

I said, I think it has a bunch of benefits and I don't see major

drawbacks

(maybe you do?).

What do you think?

Fabian

Am Do., 19. Sept. 2019 um 11:24 Uhr schrieb Fabian Hueske <
fhue...@gmail.com

:
Hi everyone,

I thought again about option #1 and something that I don't like is

that

the resolved address of xyz is different in "CREATE FUNCTION xyz"

and

"CREATE TEMPORARY FUNCTION xyz".
IMO, adding the keyword "TEMPORARY" should only change the

lifecycle of

the function, but not where it is located. 

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

[Dawid Wysakowicz]

I also like the 'System' keyword. I think we can assume we reached
consensus on this topic.

On Sat, 21 Sep 2019, 06:37 Xuefu Z,  wrote:

> +1 for using the keyword "SYSTEM". Thanks to Timo for chiming in!
>
> --Xuefu
>
> On Fri, Sep 20, 2019 at 3:28 PM Timo Walther  wrote:
>
> > Hi everyone,
> >
> > sorry, for the late replay. I give also +1 for option #2. Thus, I guess
> > we have a clear winner.
> >
> > I would also like to find a better keyword/syntax for this statement.
> > Esp. the BUILTIN keyword can confuse people, because it could be written
> > as BUILTIN, BUILDIN, BUILT_IN, or BUILD_IN. And we would need to
> > introduce a new reserved keyword in the parser which affects also
> > non-DDL queries. How about:
> >
> > CREATE TEMPORARY SYSTEM FUNCTION xxx
> >
> > The SYSTEM keyword is already a reserved keyword and in FLIP-66 we are
> > discussing to prefix some of the function with a SYSTEM_ prefix like
> > SYSTEM_WATERMARK. Also SQL defines syntax like "FOR SYSTEM_TIME AS OF".
> >
> > What do you think?
> >
> > Thanks,
> > Timo
> >
> >
> > On 20.09.19 05:45, Bowen Li wrote:
> > > Another reason I prefer "CREATE TEMPORARY BUILTIN FUNCTION" over "ALTER
> > > BUILTIN FUNCTION xxx TEMPORARILY" is - what if users want to drop the
> > > temporary built-in function in the same session? With the former one,
> > they
> > > can run something like "DROP TEMPORARY BUILTIN FUNCTION"; With the
> latter
> > > one, I'm not sure how users can "restore" the original builtin function
> > > easily from an "altered" function without introducing further
> nonstandard
> > > SQL syntax.
> > >
> > > Also please pardon me as I realized using net may not be a good idea...
> > I'm
> > > trying to fit this vote into cases listed in Flink Bylaw [1].
> > >
> > > >From the following result, the majority seems to be #2 too as it has
> the
> > > most approval so far and doesn't have strong "-1".
> > >
> > > #1：3 (+1), 1 (0), 4(-1)
> > > #2：4(0), 3 (+1), 1(+0.5)
> > > * Dawid -1/0 depending on keyword
> > > #3：2(+1), 3(-1), 3(0)
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
> > >
> > > On Thu, Sep 19, 2019 at 10:30 AM Bowen Li  wrote:
> > >
> > >> Hi,
> > >>
> > >> Thanks everyone for your votes. I summarized the result as following:
> > >>
> > >> #1：3 (+1), 1 (0), 4(-1) - net: -1
> > >> #2：4(0), 2 (+1), 1(+0.5)  - net: +2.5
> > >>  Dawid -1/0 depending on keyword
> > >> #3：2(+1), 3(-1), 3(0)   - net: -1
> > >>
> > >> Given the result, I'd like to change my vote for #2 from 0 to +1, to
> > make
> > >> it a stronger case with net +3.5. So the votes so far are:
> > >>
> > >> #1：3 (+1), 1 (0), 4(-1) - net: -1
> > >> #2：4(0), 3 (+1), 1(+0.5)  - net: +3.5
> > >>  Dawid -1/0 depending on keyword
> > >> #3：2(+1), 3(-1), 3(0)   - net: -1
> > >>
> > >> What do you think? Do you think we can conclude with this result? Or
> > would
> > >> you like to take it as a formal FLIP vote with 3 days voting period?
> > >>
> > >> BTW, I'd prefer "CREATE TEMPORARY BUILTIN FUNCTION" over "ALTER
> BUILTIN
> > >> FUNCTION xxx TEMPORARILY" because
> > >> 1. the syntax is more consistent with "CREATE FUNCTION" and "CREATE
> > >> TEMPORARY FUNCTION"
> > >> 2. "ALTER BUILTIN FUNCTION xxx TEMPORARILY" implies it alters a
> built-in
> > >> function but it actually doesn't, the logic only creates a temp
> function
> > >> with higher priority than that built-in function in ambiguous
> resolution
> > >> order; and it would behave inconsistently with "ALTER FUNCTION".
> > >>
> > >>
> > >>
> > >> On Thu, Sep 19, 2019 at 2:58 AM Fabian Hueske 
> > wrote:
> > >>
> > >>> I agree, it's very similar from the implementation point of view and
> > the
> > >>> implications.
> > >>>
> > >>> IMO, the difference is mostly on the mental model for the user.
> > >>> Instead of having a special class of temporary functions that have
> > >>> precedence over builtin functions it suggests to temporarily change
> > >>> built-in functions.
> > >>>
> > >>> Fabian
> > >>>
> > >>> Am Do., 19. Sept. 2019 um 11:52 Uhr schrieb Kurt Young <
> > ykt...@gmail.com
> >  :
> >  Hi Fabian,
> > 
> >  I think it's almost the same with #2 with different keyword:
> > 
> >  CREATE TEMPORARY BUILTIN FUNCTION xxx
> > 
> >  Best,
> >  Kurt
> > 
> > 
> >  On Thu, Sep 19, 2019 at 5:50 PM Fabian Hueske 
> > >>> wrote:
> > > Hi,
> > >
> > > I thought about it a bit more and think that there is some good
> value
> > >>> in
> >  my
> > > last proposal.
> > >
> > > A lot of complexity comes from the fact that we want to allow
> > >>> overriding
> > > built-in functions which are differently addressed as other
> functions
> >  (and
> > > db objects).
> > > We could just have "CREATE TEMPORARY FUNCTION" do exactly the same
> > >>> thing
> >  as
> > > "CREATE FUNCTION" and treat both functions exactly the same except
> > 

[jira] [Created] (FLINK-14159) flink rocksdb StreamCompressionDecorator not right

[jackylau (Jira)]

jackylau created FLINK-14159:


 Summary: flink rocksdb StreamCompressionDecorator not right
 Key: FLINK-14159
 URL: https://issues.apache.org/jira/browse/FLINK-14159
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Affects Versions: 1.9.0
Reporter: jackylau
 Fix For: 1.10.0


I think the current flink rocksdb StreamCompressionDecorator is not right 
calling method 

getCompressionDecorator(executionConfig) which defalut value is false.That is 
to say, current compression is none.But I find rocksdb  using 
{{options.compression}} to specify the compression to use. By default it is 
Snappy, which you can see here 
[https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide]. And I use 
rocksdb tool sstdump to find it is indeed snappy compression.

So I think it should be return SnappyStreamCompressionDecorator.INSTANCE  
rather than getCompressionDecorator( executionConfig) 

Coud i commit a PR?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

[Bowen Li]

"SYSTEM" sounds good to me. FYI, this FLIP only impacts low level of the
SQL function stack and won't actually involve any DDL, thus I will just
document the decision and we should keep it in mind when it's time to
implement the DDLs.

I'm in the process of updating the FLIP to reflect changes required for
option #2, will send a new version for review soon.



On Fri, Sep 20, 2019 at 4:02 PM Dawid Wysakowicz 
wrote:

> I also like the 'System' keyword. I think we can assume we reached
> consensus on this topic.
>
> On Sat, 21 Sep 2019, 06:37 Xuefu Z,  wrote:
>
> > +1 for using the keyword "SYSTEM". Thanks to Timo for chiming in!
> >
> > --Xuefu
> >
> > On Fri, Sep 20, 2019 at 3:28 PM Timo Walther  wrote:
> >
> > > Hi everyone,
> > >
> > > sorry, for the late replay. I give also +1 for option #2. Thus, I guess
> > > we have a clear winner.
> > >
> > > I would also like to find a better keyword/syntax for this statement.
> > > Esp. the BUILTIN keyword can confuse people, because it could be
> written
> > > as BUILTIN, BUILDIN, BUILT_IN, or BUILD_IN. And we would need to
> > > introduce a new reserved keyword in the parser which affects also
> > > non-DDL queries. How about:
> > >
> > > CREATE TEMPORARY SYSTEM FUNCTION xxx
> > >
> > > The SYSTEM keyword is already a reserved keyword and in FLIP-66 we are
> > > discussing to prefix some of the function with a SYSTEM_ prefix like
> > > SYSTEM_WATERMARK. Also SQL defines syntax like "FOR SYSTEM_TIME AS OF".
> > >
> > > What do you think?
> > >
> > > Thanks,
> > > Timo
> > >
> > >
> > > On 20.09.19 05:45, Bowen Li wrote:
> > > > Another reason I prefer "CREATE TEMPORARY BUILTIN FUNCTION" over
> "ALTER
> > > > BUILTIN FUNCTION xxx TEMPORARILY" is - what if users want to drop the
> > > > temporary built-in function in the same session? With the former one,
> > > they
> > > > can run something like "DROP TEMPORARY BUILTIN FUNCTION"; With the
> > latter
> > > > one, I'm not sure how users can "restore" the original builtin
> function
> > > > easily from an "altered" function without introducing further
> > nonstandard
> > > > SQL syntax.
> > > >
> > > > Also please pardon me as I realized using net may not be a good
> idea...
> > > I'm
> > > > trying to fit this vote into cases listed in Flink Bylaw [1].
> > > >
> > > > >From the following result, the majority seems to be #2 too as it has
> > the
> > > > most approval so far and doesn't have strong "-1".
> > > >
> > > > #1：3 (+1), 1 (0), 4(-1)
> > > > #2：4(0), 3 (+1), 1(+0.5)
> > > > * Dawid -1/0 depending on keyword
> > > > #3：2(+1), 3(-1), 3(0)
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
> > > >
> > > > On Thu, Sep 19, 2019 at 10:30 AM Bowen Li 
> wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> Thanks everyone for your votes. I summarized the result as
> following:
> > > >>
> > > >> #1：3 (+1), 1 (0), 4(-1) - net: -1
> > > >> #2：4(0), 2 (+1), 1(+0.5)  - net: +2.5
> > > >>  Dawid -1/0 depending on keyword
> > > >> #3：2(+1), 3(-1), 3(0)   - net: -1
> > > >>
> > > >> Given the result, I'd like to change my vote for #2 from 0 to +1, to
> > > make
> > > >> it a stronger case with net +3.5. So the votes so far are:
> > > >>
> > > >> #1：3 (+1), 1 (0), 4(-1) - net: -1
> > > >> #2：4(0), 3 (+1), 1(+0.5)  - net: +3.5
> > > >>  Dawid -1/0 depending on keyword
> > > >> #3：2(+1), 3(-1), 3(0)   - net: -1
> > > >>
> > > >> What do you think? Do you think we can conclude with this result? Or
> > > would
> > > >> you like to take it as a formal FLIP vote with 3 days voting period?
> > > >>
> > > >> BTW, I'd prefer "CREATE TEMPORARY BUILTIN FUNCTION" over "ALTER
> > BUILTIN
> > > >> FUNCTION xxx TEMPORARILY" because
> > > >> 1. the syntax is more consistent with "CREATE FUNCTION" and "CREATE
> > > >> TEMPORARY FUNCTION"
> > > >> 2. "ALTER BUILTIN FUNCTION xxx TEMPORARILY" implies it alters a
> > built-in
> > > >> function but it actually doesn't, the logic only creates a temp
> > function
> > > >> with higher priority than that built-in function in ambiguous
> > resolution
> > > >> order; and it would behave inconsistently with "ALTER FUNCTION".
> > > >>
> > > >>
> > > >>
> > > >> On Thu, Sep 19, 2019 at 2:58 AM Fabian Hueske 
> > > wrote:
> > > >>
> > > >>> I agree, it's very similar from the implementation point of view
> and
> > > the
> > > >>> implications.
> > > >>>
> > > >>> IMO, the difference is mostly on the mental model for the user.
> > > >>> Instead of having a special class of temporary functions that have
> > > >>> precedence over builtin functions it suggests to temporarily change
> > > >>> built-in functions.
> > > >>>
> > > >>> Fabian
> > > >>>
> > > >>> Am Do., 19. Sept. 2019 um 11:52 Uhr schrieb Kurt Young <
> > > ykt...@gmail.com
> > >  :
> > >  Hi Fabian,
> > > 
> > >  I think it's almost the same with #2 with different keyword:
> > > 
> > >  CREATE TEMPORARY BUILTIN FUNCTION xxx
> 

Re: Add Bucket File System Table Sink

[Kurt Young]

Hi Jun,

Thanks for your understanding. If we all agree adding this functionality
into FLIP-63 is
a good idea, I would suggest you also help reviewing the FLIP-63 design
document to
see if current design meet your requirements. You can also raise some
comments to
the design document if you have some thoughts.

Best,
Kurt


On Fri, Sep 20, 2019 at 10:44 PM Jun Zhang <825875...@qq.com> wrote:

> Hi，Fabian ：
>
> Thank you very much for your suggestion. This is when I use flink sql to
> write data to hdfs at work. I feel that it is inconvenient. I wrote this
> function, and then I want to contribute it to the community. This is my
> first PR , some processes may not be clear, I am very sorry.
>
> Kurt suggested combining this feature with FLIP-63 because they have some
> common features, such as write data to file system with kinds of format, so
> I want to treat this function as a sub-task of FLIP-63. Add a partitionable
>  bucket file system table sink.
>
> I then added the document and sent a DISCUSS to explain my detailed design
> ideas and implementation. How do you see it?
>
>
>
> -- Original --
> *From:* Fabian Hueske 
> *Date:* Fri,Sep 20,2019 9:38 PM
> *To:* Jun Zhang <825875...@qq.com>
> *Cc:* dev , user 
> *Subject:* Re: Add Bucket File System Table Sink
>
> Hi Jun,
>
> Thank you very much for your contribution.
>
> I think a Bucketing File System Table Sink would be a great addition.
>
> Our code contribution guidelines [1] recommend to discuss the design with
> the community before opening a PR.
> First of all, this ensures that the design is aligned with Flink's codebase
> and the future features.
> Moreover, it helps to find a committer who can help to shepherd the PR.
>
> Something that is always a good idea is to split a contribution in multiple
> smaller PRs (if possible).
> This allows for faster review and progress.
>
> Best, Fabian
>
> [1] https://flink.apache.org/contributing/contribute-code.html
>
> Am Di., 17. Sept. 2019 um 04:39 Uhr schrieb Jun Zhang <825875...@qq.com>:
>
> > Hello everyone:
> > I am a user and fan of flink. I also want to join the flink community. I
> > contributed my first PR a few days ago. Can anyone help me to review my
> > code? If there is something wrong, hope I would be grateful if you can
> give
> > some advice.
> >
> > This PR is mainly in the process of development, I use sql to read data
> > from kafka and then write to hdfs, I found that there is no suitable
> > tablesink, I found the document and found that File System Connector is
> > only experimental (
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/connect.html#file-system-connector
> ),
> > so I wrote a Bucket File System Table Sink that supports writing stream
> > data. Hdfs, file file system, data format supports json, csv, parquet,
> > avro. Subsequently add other format support, such as protobuf, thrift,
> etc.
> >
> > In addition, I also added documentation, python api, units test,
> > end-end-test, sql-client, DDL, and compiled on travis.
> >
> > the issue is https://issues.apache.org/jira/browse/FLINK-12584
> > thank you very much
> >
> >
> >
>
>

Re: [FLINK-14076] non-deserializable root cause in DeclineCheckpoint

[Jeffrey Martin]

To be clear -- I'm happy to make a PR for either option below. (Either is
<10 lines diff.) It's just the contributor guidelines said to get consensus
first and then only make a PR if I'm assigned to do the work.

On Fri, Sep 20, 2019 at 12:23 PM Jeffrey Martin 
wrote:

> (possible dupe; I wasn't subscribed before and the previous message didn't
> seem to go through)
>
> I'm on Flink v1.9 with the Kafka connector and a standalone JM.
> If FlinkKafkaProducer fails while checkpointing, it throws a
> KafkaException which gets wrapped in a CheckpointException which is sent to
> the JM as a DeclineCheckpoint. KafkaException isn't on the JM default
> classpath, so the JM throws a fairly cryptic ClassNotFoundException. The
> details of the KafkaException wind up suppressed so it's impossible to
> figure out what actually went wrong.
>
> I can think of two fixes that would prevent this from occurring in the
> Kafka or other connectors in the future:
> 1. DeclineCheckpoint should always send a SerializedThrowable to the JM
> rather than allowing CheckpointExceptions with non-deserializable root
> causes to slip through
> 2. CheckpointException should always capture its wrapped exception as a
> SerializedThrowable (i.e., use 'super(new SerializedThrowable(cause))'
> rather than 'super(cause)').
>
> Thoughts?
>

[jira] [Created] (FLINK-14150) Unnecessary __pycache__ directories appears in pyflink.zip

[Wei Zhong (Jira)]

Wei Zhong created FLINK-14150:
-

 Summary: Unnecessary __pycache__ directories appears in pyflink.zip
 Key: FLINK-14150
 URL: https://issues.apache.org/jira/browse/FLINK-14150
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.9.0, 1.10.0
Reporter: Wei Zhong


It seems we are packaging __pycache__ directories into pyflink.zip. These 
directories contain bytecode cache files that are automatically generated by 
python3. We should remove them from the python source code folder before 
packaging.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Add ARM CI build to Flink (information-only)

[Xiyuan Wang]

Sure,  we can run daily ARM job as Travis CI nightly jobs firstly. Once
it's stable enough, we can consider adding it to peer PR.

BTW, I tested flink-end-to-end-test on ARM in last few days. Keeping the
same as Travis, all 7 scenarios were tested:

1. split_checkpoints.sh
2. split_sticky.sh
3. split_ha.sh
4. split_heavy.sh
5. split_misc_hadoopfree.sh
6. split_misc.sh
7. split_container.sh

The 1st-6th scenarios works well within some hacking and bug fixing locally:
1. frocksdb doesn't have official ARM release, so I built and install
it locally for ARM.
  https://issues.apache.org/jira/browse/FLINK-13598
2. Prometheus has ARM release but the test always download x86 version.
Download the correct version can fix the issue.
  https://issues.apache.org/jira/browse/FLINK-14086
3. Elasticsearch 6.0+ enables Xpack machine learning feature by
default, but this feature doesn't support ARM. So Elasticsearch 6.0+ failed
to start on ARM. Set `Xpack.ml.enabled: false` can fix this issue.
  https://issues.apache.org/jira/browse/FLINK-14126

The 7th scenario for container failed because:
1. docker-compose doesn't have official ARM package. Use `apt install
docker-compose` can solve the problem.
2. minikube doesn't support ARM arch. Use kubeadm for K8S installation
can solve the problem.

Fixing the problem mentioned above is not hard. So I think we can add flink
build, unit-test and e2e test as nightly jobs now.

Any idea?

Thanks.

Stephan Ewen  于2019年9月19日周四 下午5:44写道：

> My gut feeling is that having a CI that only runs on a specific command
> will not help too much.
>
> What about going with nightly builds then? We could set up the ARM CI the
> same way as the Travis CI nightly builds (cron builds). They report build
> failures to "bui...@flink.apache.org".
> Maybe Chesnay or Jark could help with what needs to be done to post to that
> mailing list?
>
> A requirement would be that the builds are stable, from the ARM
> perspective, meaning that there are no failures at the moment caused by ARM
> specific issue.
>
> What do the others think?
>
>
> On Tue, Sep 3, 2019 at 4:40 AM Xiyuan Wang 
> wrote:
>
> > The ARM CI trigger has been changed to `github comment` way only. It
> means
> > that every PR won't start ARM test unless a comment `check_arm` is added.
> > Like what I did in the PR[1].
> >
> > A POC for Flink nightly end to end test job is created as well[2]. I'll
> > improve it then.
> >
> > Any feedback or question?
> >
> >
> > [1]: https://github.com/apache/flink/pull/9416
> >  https://github.com/apache/flink/pull/9416#issuecomment-527268203
> > [2]: https://github.com/theopenlab/openlab-zuul-jobs/pull/631
> >
> >
> > Thanks
> >
> > Xiyuan Wang  于2019年8月26日周一 下午7:41写道：
> >
> > > Before ARM CI is ready, I can close the CI test for each PR and let it
> > > only be triggered by PR comment.  It's quite easy for OpenLab to do
> this.
> > >
> > > OpenLab have many job piplines[1].  Now I use `check` pipline in
> > > https://github.com/apache/flink/pull/9416. The job trigger contains
> > > github_action and github_comment[2]. I can create a new pipline for
> > Flink,
> > > the new trigger can only contain github_coment like:
> > >
> > > trigger:
> > >   github:
> > >  - event: pull_request
> > >action: comment
> > >comment: (?i)^\s*recheck_arm_build\s*$
> > >
> > > So that the ARM job will not be ran for every PR. It'll be just ran for
> > > the PR which have `recheck_arm_build` comment.
> > >
> > > Then once ARM CI is ready, I can add it back.
> > >
> > >
> > > nightly tests can be added as well of couse. There is a kind of job in
> > > OpenLab called `periodic job`. We can use it for Flink daily nightly
> > tests.
> > > If any error occur, the report can be sent to bui...@flink.apache.org
> > as
> > > well.
> > >
> > > [1]:
> > >
> >
> https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/pipelines.yaml
> > > [2]:
> > >
> >
> https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/pipelines.yaml#L10-L19
> > >
> > > Stephan Ewen  于2019年8月26日周一 下午6:13写道：
> > >
> > >> Adding CI builds for ARM makes only sense when we actually take them
> > into
> > >> account as "blocking a merge", otherwise there is no point in having
> > them.
> > >> So we would need to be prepared to do that.
> > >>
> > >> The cases where something runs in UNIX/x64 but fails on ARM are few
> > cases
> > >> and so far seem to have been related to libraries or some magic that
> > tries
> > >> to do system dependent actions outside Java.
> > >>
> > >> One worthwhile discussion could be whether to run the ARM CI builds as
> > >> part
> > >> of the nightly tests, not on every commit.
> > >> There are a lot of nightly tests, for example for different Java /
> > Scala /
> > >> Hadoop versions.
> > >>
> > >> On Mon, Aug 26, 2019 at 10:46 AM Xiyuan Wang <
> wangxiyuan1...@gmail.com>
> > >> wrote:
> > >>
> > >> > Sorry, maybe my words is misleading.
> > >> >
> > >> > We are just 

[jira] [Created] (FLINK-14139) Fix potential memory leak of rest server when using session/standalone cluster

[Yingjie Cao (Jira)]

Yingjie Cao created FLINK-14139:
---

 Summary: Fix potential memory leak of rest server when using 
session/standalone cluster
 Key: FLINK-14139
 URL: https://issues.apache.org/jira/browse/FLINK-14139
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / REST
Reporter: Yingjie Cao
 Fix For: 1.10.0


Flink's rest server uses netty decoder for http request processing and file 
uploading. However io.netty.handler.codec.http.multipart.DiskAttribute and 
io.netty.handler.codec.http.multipart.DiskFileUpload class of netty would 
register some temp files, including post chunks and upload file chunks, to 
java.io.DeleteOnExitHook which has a potential of memory leak, because the 
registered file names will never be deleted before the cluster stops. 

Most of the time, this is not a big problem, however we use Flink session 
cluster a long running service for ad-hoc SQL query, this problem gets worse.

In fact, Flink handles the clean up of temp files through 
org.apache.flink.util.ShutdownHookUtil (though not including the post chunks), 
so it is no need to register these files to java.io.DeleteOnExitHook.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14140) The Flink Logo Displayed in Flink Python Shell is Broken

[Wei Zhong (Jira)]

Wei Zhong created FLINK-14140:
-

 Summary: The Flink Logo Displayed in Flink Python Shell is Broken
 Key: FLINK-14140
 URL: https://issues.apache.org/jira/browse/FLINK-14140
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.9.0, 1.10.0
Reporter: Wei Zhong
 Attachments: image-2019-09-20-15-03-09-111.png

when executing "./pyflink-shell.sh local"，we can get such a Logo:

!image-2019-09-20-15-03-09-111.png|width=396,height=553!

It seems that the upper right corner of the squirrel is broken.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14143) Failed Attempt does not display in the timeline

[Yadong Xie (Jira)]

Yadong Xie created FLINK-14143:
--

 Summary: Failed Attempt does not display in the timeline
 Key: FLINK-14143
 URL: https://issues.apache.org/jira/browse/FLINK-14143
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Web Frontend
Reporter: Yadong Xie
 Attachments: 屏幕快照 2019-09-20 下午3.46.40.png

There may exist more than one attempt in a subtask, but in the timeline page, 
the Web UI can only get and visualize the latest execution attempt of a subtask 
timeline, there is no way to get a failed attempt timeline in current REST API.

!屏幕快照 2019-09-20 下午3.46.40.png|width=453,height=207!

 

REST API needed:

add failed attempt time in /jobs/:jobid/vertices/:vertexid/subtasktimes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14145) getLatestCheckpoint returns wrong checkpoint

[Ufuk Celebi (Jira)]

Ufuk Celebi created FLINK-14145:
---

 Summary: getLatestCheckpoint returns wrong checkpoint
 Key: FLINK-14145
 URL: https://issues.apache.org/jira/browse/FLINK-14145
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.9.0
Reporter: Ufuk Celebi


The flag to prefer checkpoints for recovery introduced in FLINK-11159 breaks 
returns the wrong checkpoint as the latest one if enabled.

The current implementation assumes that the latest checkpoint is a savepoint 
and skips over it. I attached a patch for 
{{StandaloneCompletedCheckpointStoreTest}} that demonstrates the issue.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14146) Introduce Pulsar connector

[Yijie Shen (Jira)]

Yijie Shen created FLINK-14146:
--

 Summary: Introduce Pulsar connector
 Key: FLINK-14146
 URL: https://issues.apache.org/jira/browse/FLINK-14146
 Project: Flink
  Issue Type: New Feature
Reporter: Yijie Shen


Please see FLIP-72 for detailed information:

[https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14141) Locate Jobs/SubTask/Vertex Running on TaskManager

[Yadong Xie (Jira)]

Yadong Xie created FLINK-14141:
--

 Summary: Locate Jobs/SubTask/Vertex Running on TaskManager
 Key: FLINK-14141
 URL: https://issues.apache.org/jira/browse/FLINK-14141
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Web Frontend
Reporter: Yadong Xie
 Attachments: 屏幕快照 2019-09-20 下午3.20.05.png

As we know, the subtask associated with job vertex is running on task manager, 
but in current design in Web UI, there is no way to get the associated jobs, 
vertex or subtask information from the task manager.


We should add a Job List tab in the task manager page, giving a list of 
job/vertex/subtask associated with the current task manager, with FLINK-13894 
users can link subtask and taskmanger together when troubleshooting. Here is a 
demo below.

!屏幕快照 2019-09-20 下午3.20.05.png|width=602,height=240!

REST API needed:

add /taskmanagers/:taskmanagerid/jobs API to get associated jobs with current 
taskmanager.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] FLIP-72: Introduce Pulsar Connector

[Yijie Shen]

Hi everyone,

Per discussion in the previous thread
,
I have created FLIP-72 to kick off a more detailed discussion on the Flink
Pulsar connector:

https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector

In short, the connector has the following features:

   -

   Pulsar as a streaming source with exactly-once guarantee.
   -

   Sink streaming results to Pulsar with at-least-once semantics.
   -

   Build upon Flink new Table API Type system (FLIP-37
   

   ), and can automatically (de)serialize messages with the help of Pulsar
   schema.
   -

   Integrate with Flink new Catalog API (FLIP-30
   
),
   which enables the use of Pulsar topics as tables in Table API as well as
   SQL client.


https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#heading=h.28v5v23yeq1u


Would love to here your thoughts on this.

Best,
Yijie

[jira] [Created] (FLINK-14142) Add more metrics to task manager list

[Yadong Xie (Jira)]

Yadong Xie created FLINK-14142:
--

 Summary: Add more metrics to task manager list
 Key: FLINK-14142
 URL: https://issues.apache.org/jira/browse/FLINK-14142
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Web Frontend
Reporter: Yadong Xie
 Attachments: 屏幕快照 2019-09-20 下午3.31.58.png

In the task manager list page, besides the free slots and all slots, we could 
add memory and CPU usage metrics to the taskmanager, these metrics are already 
available in blink.

!屏幕快照 2019-09-20 下午3.31.58.png|width=619,height=266!

 

REST API needed:

add CPU and memory usage metrics in the /taskmanagers API



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14144) Add Metric for job manager

[Yadong Xie (Jira)]

Yadong Xie created FLINK-14144:
--

 Summary: Add Metric for job manager
 Key: FLINK-14144
 URL: https://issues.apache.org/jira/browse/FLINK-14144
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Web Frontend
Reporter: Yadong Xie


There is a metric tab in each taskmanager page, but there is no metric data in 
the job manager, it would be better for users to monitor the cluster with job 
manager metrics.

The 
[API|https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobmanager-metrics]
 provided now could only get the metric key without a value.

REST API needed:

add metrics value in /jobmanager/metrics API



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Confluence permission for FLIP creation

[Terry Wang]

Thanks, Fabian

Best,
Terry Wang



> 在 2019年9月19日，下午7:37，Fabian Hueske  写道：
> 
> Hi Terry,
> 
> I gave you permissions.
> 
> Thanks, Fabian
> 
> Am Do., 19. Sept. 2019 um 04:09 Uhr schrieb Terry Wang :
> 
>> Hi all,
>> 
>> As communicated in an email thread, I'm proposing Flink SQL ddl
>> enhancement. I have a draft design doc that I'd like to convert it to a
>> FLIP. Thus, it would be great if anyone who can grant me the write access
>> to Confluence. My Confluence ID is zjuwangg.
>> 
>> It would be nice if any of you can help on this.
>> 
>> Best,
>> Terry Wang
>> 
>> 
>> 
>> 

[jira] [Created] (FLINK-14148) Investigate pushing predicate/projection to underlying Hive input format

[Rui Li (Jira)]

Rui Li created FLINK-14148:
--

 Summary: Investigate pushing predicate/projection to underlying 
Hive input format
 Key: FLINK-14148
 URL: https://issues.apache.org/jira/browse/FLINK-14148
 Project: Flink
  Issue Type: Sub-task
Reporter: Rui Li






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14149) Introduce ZooKeeperLeaderElectionServiceNG

[tison (Jira)]

tison created FLINK-14149:
-

 Summary: Introduce ZooKeeperLeaderElectionServiceNG
 Key: FLINK-14149
 URL: https://issues.apache.org/jira/browse/FLINK-14149
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: tison
Assignee: tison


Subsequent to the discussion in FLINK-10333, we reach a consensus that refactor 
ZK based storage with a transaction store mechanism. The overall design can be 
found in the design document linked below.

This subtask is aimed at introducing the prerequisite to adopt transaction 
store, i.e., a new leader election service for ZK scenario. The necessity is 
that we have to retrieve the corresponding latch path per contender following 
the algorithm describe in FLINK-10333.

Here is the (descriptive) details about the implementation.

We adopt the optimized version of [this 
recipe|https://zookeeper.apache.org/doc/current/recipes.html#sc_leaderElection][1].
 Code details can be found in [this 
branch|https://github.com/TisonKun/flink/tree/election-service] and the state 
machine can be found in the design document attached. Here is only the most 
important two differences from the former implementation:

(1) *Leader election is an one-shot service.*

Specifically, we only create one latch for a specific contender. We tolerate 
{{SUSPENDED}} a.k.a. {{CONNECTIONLOSS}} so that the only situation we lost 
leadership is session expired, which infers the ephemeral latch znode is 
deleted. We don't re-participant as contender so after {{revokeLeadership}} a 
contender will never be granted any more. This is not a problem but we can do 
further refactor in contender side for better behavior.

(2) *Leader info znode is {{PERSISTENT}}.*

It is because we now regard create/setData to leader info znode a leader-only 
operation and thus do it in a transaction. If we keep using ephemeral znode it 
is hard to test. Because we share ZK client so the ephemeral znode is not 
deleted so that we should deal with complex znode stat that transaction cannot 
simply deal with. And since znode is {{PERSISTENT}} we introduce a 
{{concealLeaderInfo}} method called back on contender stop to clean up.

Another topic is about interface. Back to the big picture of FLINK-10333 we 
eventually use a transaction store for persisting job graph and checkpoint and 
so on. So there will be a {{getLeaderStore}} method added on 
{{LeaderElectionServices}}. Because we don't use it at all it is an open 
question that whether we add the method to the interface in this subtask. And 
if so, whether we implement it for other election services implementation.

{{concealLeaderInfo}} is another method appeared in the document that aimed at 
clean up leader info node on stop. So the same problem as {{getLeaderStore}}.

**For what we gain**

1. Basics for the overall goal under FLINK-10333
2. Leader info node must be modified by the current leader. Thus we can reduce 
a lot of concurrency handling logic in currently ZLES, including using 
{{NodeCache}} as well as dealing with complex stat of ephemeral leader info 
node.

[1] For other implementation, I start [a 
thread|https://lists.apache.org/x/thread.html/594b66ecb1d60b560a5c4c08ed1b2a67bc29143cb4e8d368da8c39b2@%3Cuser.zookeeper.apache.org%3E]
 in ZK and Curator to discuss. Anyway, it will be implementation details only, 
and interfaces and semantics should not be affected.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14152) Add class for DocCountVectorizerMapper.

[Xu Yang (Jira)]

Xu Yang created FLINK-14152:
---

 Summary: Add  class  for  DocCountVectorizerMapper.
 Key: FLINK-14152
 URL: https://issues.apache.org/jira/browse/FLINK-14152
 Project: Flink
  Issue Type: Sub-task
  Components: Library / Machine Learning
Reporter: Xu Yang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14151) Add class for DocHashIDFVectorizerModelMapper.

[Xu Yang (Jira)]

Xu Yang created FLINK-14151:
---

 Summary: Add  class  for  DocHashIDFVectorizerModelMapper.
 Key: FLINK-14151
 URL: https://issues.apache.org/jira/browse/FLINK-14151
 Project: Flink
  Issue Type: Sub-task
  Components: Library / Machine Learning
Reporter: Xu Yang


DocHashIDFVectorizerModelMapper  is  a  transformer  to  transform  a  document 
 to  a  sparse  vector  based  on  the  inverse  document  frequency(idf).
 * Add  DocHashIDFVectorizerModelMapper  for  the  operation  of  the  
DocHashIDFVectorizerModelMapper.
 * Add  DocHashIDFVectorizerModelData  to  save  model  data.
 * Add  DocHashIDFVectorizerModelDataConverter  to  serialize  and  deserialize 
 model.
 * Add  DocHashIDFVectorizerPredictParams  for  the  params  of  
DocHashIDFVectorizerModelMapper.
 * Add  DocHashIDFVectorizerModelMapperTest  for  the  test  example.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: java8 lambdas and exceptions lead to compile error

[zz]

thanks for reply. "add some context/comment" is very necessary, but I am
not sure where to add to remind others for avoiding similar mistakes, so Is
that a better way to add corresponding grammar checkstyle
in checkstyle.xml? we can remove corresponding grammar checkstyle when we
upgrade new Java version.In this way others committers can
avoid similar problem.

Till Rohrmann  于2019年9月19日周四 下午3:37写道：

> Hi,
>
> if there is an easy way to make it also work with Java 1.8.0_77 I guess we
> could change it. That way we would make the life of our users easier.
>
> The solution proposed by JDK-8054569 seems quite simple. The only downside
> I see is that it could easily fell victim of a future refactoring/clean up
> if we don't add some context/comment why the explicit type has been
> introduced. Alternatively, we could state on the website which Java version
> you need to build Flink.
>
> Cheers,
> Till
>
> On Thu, Sep 19, 2019 at 8:53 AM zz  wrote:
>
> > Hey all,
> > Recently, I used flink to do secondary development, when compile flink
> > master(up-to-date) by using Java 1.8.0_77, got errors as follow:
> >
> > compile (default-compile) on project flink-table-api-java: Compilation
> > failure
> >
> >
> /home/*/zzsmdfj/sflink/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/utils/factories/Cal
> > culatedTableFactory.java:[90,53] unreported exception X; must be caught
> or
> > declared to be thrownat
> > org.apache.maven.lifecycle.internal.MojoExecutor.execute
> > (MojoExecutor.java:213)
> > at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> > (MojoExecutor.java:154)
> > at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> > (MojoExecutor.java:146)
> > at
> > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
> > (LifecycleModuleBuilder.java:117)
> > at
> > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
> > (LifecycleModuleBuilder.java:81)
> > at
> >
> >
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
> > (SingleThreadedBuilder.java:51)
> > at org.apache.maven.lifecycle.internal.LifecycleStarter.execute
> > (LifecycleStarter.java:128)
> > at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
> > at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
> > at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
> > at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
> > at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
> > at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke
> > (NativeMethodAccessorImpl.java:62)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke
> > (DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke (Method.java:498)
> > at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
> > (Launcher.java:289)
> > at org.codehaus.plexus.classworlds.launcher.Launcher.launch
> > (Launcher.java:229)
> > at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
> > (Launcher.java:415)
> > at org.codehaus.plexus.classworlds.launcher.Launcher.main
> > (Launcher.java:356)
> > Caused by: org.apache.maven.plugin.compiler.CompilationFailureException:
> > Compilation failure
> >
> > if using Java 1.8.0_102 to compile, it build success. it maybe a case of
> > bug JDK-8054569 .
> >
> > Is that a problem？and What should I do with this？any comments would be
> > appreciated.
> >
> > issue：https://issues.apache.org/jira/browse/FLINK-14093
> >
>

[jira] [Created] (FLINK-14153) Add to BLAS a method that performs DenseMatrix and SparseVector multiplication.

[Xu Yang (Jira)]

Xu Yang created FLINK-14153:
---

 Summary: Add to BLAS a method that performs DenseMatrix and 
SparseVector multiplication.
 Key: FLINK-14153
 URL: https://issues.apache.org/jira/browse/FLINK-14153
 Project: Flink
  Issue Type: Sub-task
  Components: Library / Machine Learning
Reporter: Xu Yang


Previously there is "gemv" method in BLAS that performs multiplications between 
DenseMatrix and DenseVector. Here we add another one that performs 
multiplications between DenseMatrix and SparseVector.
 * Add gemv method to BLAS.
 * Add BLASTest that provides test cases for BLAS.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14154) Add the class for multivariate Gaussian Distribution.

[Xu Yang (Jira)]

Xu Yang created FLINK-14154:
---

 Summary: Add the class for multivariate Gaussian Distribution.
 Key: FLINK-14154
 URL: https://issues.apache.org/jira/browse/FLINK-14154
 Project: Flink
  Issue Type: Sub-task
  Components: Library / Machine Learning
Reporter: Xu Yang


Multivariate Gaussian Distribution is a generalization of the one-dimensional 
(univariate) normal distribution to higher dimensions. In machine learning, 
multivariate Gaussian Distribution finds usages in algorithms link Gaussian 
Mixture Model.
 * Add MultivariateGaussian that provides basic functionality for a 
Multivariate Gaussian (Normal) Distribution.
 * Add MultivariateGaussianTest that provides test cases for 
MultivariateGaussian.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

[Sijie Guo]

Thanks Becket.

I think it is better for the Flink community to judge the benefits of doing
this. I was trying to provide some views from outsiders.

Thanks,
Sijie

On Fri, Sep 20, 2019 at 10:25 AM Becket Qin  wrote:

> Hi Sijie,
>
> Yes, we will have to support existing old connectors and new connectors in
> parallel for a while. We have to take that maintenance overhead because
> existing connectors have been used by the users for a long time. I guess It
> may take at least a year for us to fully remove the old connectors.
>
> Process wise, we can do the same for Pulsar connector. But I am not sure if
> we want to have the same burden on Pulsar connector, and I would like to
> understand the benefit of doing that.
>
> For users, the benefit of having the old Pulsar connector checked in seems
> limited because 1) that code base will be immediately deprecated in the
> next release in 3-4 months; 2) users can always use it even if it is not in
> the Flink code base. Admittedly it is not as convenient as having it in
> Flink code base, but doesn't seem super either. And after 3-4 months, users
> can just use the new connector in Flink repo.
>
> For Flink developers, the old connector code base is not something that we
> want to evolve later. Instead, these code will be deprecated and
> removed. So why do we want to get a beta version out to attract people to
> use something we don't want to maintain?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo  wrote:
>
> > Thanks everyone here. Sorry for jumping into the discussion here.
> >
> > I am not very familiar about the deprecation process in Flink. If I
> > misunderstood the process, please fix me.
> >
> > As far as I understand, FLIP-27 is introducing a new unified API for
> > connectors. After it introduces the new API
> > and before moving all the existing connectors from old API to new API,
> both
> > old ApI and new API will co-exist
> > for a while until Flink moves all existing connectors to new API. So the
> > Pulsar connector (using old API) can
> > follow the deprecation process with other connector using old API and the
> > deprecation of old API, no?
> >
> > If that's the case, I think contributing the current connector back to
> > Flink rather than maintaining it outside Flink
> > would provide a bit more benefits. We can deprecate the existing
> > streamnative/pulsar-flink repo and point the users
> > to use the connector in Flink repo. So all the review processes will
> happen
> > within Flink for both old connector and
> > new connector. It also reduces the confusions for the users as the
> > documentation and code base happen in one place.
> >
> > Thoughts?
> >
> > - Sijie
> >
> >
> >
> >
> > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin 
> wrote:
> >
> > > Thanks for the explanation, Stephan. I have a few questions / thoughts.
> > >
> > > So that means we will remove the old connector without a major version
> > > bump, is that correct?
> > >
> > > I am not 100% sure if mixing 1.10 connectors with 1.11 connectors will
> > > always work because we saw some dependency class collisions in the
> past.
> > To
> > > make it safe we may have to maintain the old code for one more release.
> > >
> > > To be honest I am still wondering if we have to put the old connector
> in
> > > Flink repo. if we check in the old connector to Flink. We will end up
> in
> > > the following situation:
> > > 1. Old connector in streamnative/pulsar-flink repo.
> > > 2. Old connector in Flink Repo, which may be different from the one in
> > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in 1.12)
> > > 3. New connector in Flink Repo.
> > >
> > > We need to think about how to make the users in each case happy.
> > > - For users of (1), I assume Sijie and Yijie will have to maintain the
> > code
> > > a bit longer for its own compatibility even after we have (2). In that
> > > case, bugs found in old connector may or may not need to be fixed in
> both
> > > Flink and the streamnative/pulsar-flink repo.
> > > - For users of (2), will we provide bug fixes? If we do, it will be a
> > > little awkward because those bug fixes will be immediately deprecated
> in
> > > 1.11, and removed in 1.12. So we are essentially asking users to
> migrate
> > > away from the bug fix. After Flink 1.12, users may still have to switch
> > to
> > > use (3) due to the potential dependency class conflicts mentioned
> above.
> > > - Users of (3) have a much easier life and don't need to worry too
> much.
> > >
> > > The above story seems a little complicated to tell. I think it will be
> > much
> > > easier to not have (2) at all.
> > > 1. Old connector in streamnative/pulsar-flink repo.
> > > 3. New connector in Flink Repo.
> > >
> > > - Old connector will only be maintained in streamnative/pulsar-flink
> repo
> > > until it is fully deprecated. Users can always use the existing Pulsar
> > > connector in that repo.
> > > - New connector will be in 

[jira] [Created] (FLINK-14155) Add a wrapper class of a JSON library to provide the unified json format.

[Xu Yang (Jira)]

Xu Yang created FLINK-14155:
---

 Summary: Add a wrapper class of a JSON library to provide the 
unified json format.
 Key: FLINK-14155
 URL: https://issues.apache.org/jira/browse/FLINK-14155
 Project: Flink
  Issue Type: Sub-task
  Components: Library / Machine Learning
Reporter: Xu Yang


There need to unify the format when the data is serialized to external storage 
and can also be deserialized conveniently. The class support the toJson and 
fromJson.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCEMENT] September 2019 Bay Area Apache Flink Meetup

[Xuefu Zhang]

Hi all,

Happy Friday!

As a kind reminder, the meetup is ON next Tuesday at Yelp HQ in San
Francisco. See you all there at 6:30pm.

Regards,
Xuefu

On Fri, Aug 30, 2019 at 11:44 AM Xuefu Zhang  wrote:

> Hi all,
>
> As promised, we planned to have quarterly Flink meetup and now it's about
> the time. Thus, I'm happy to announce that the next Bay Area Apache Flink
> Meetup [1] is scheduled on Sept. 24 at Yelp, 140 New Montgomery in San
> Francisco.
>
> Schedule:
>
> 6:30 - Door open
> 6:30 - 7:00 PM Networking and Refreshments
> 7:00 - 8:30 PM Short talks
>
> -- Two years of Flink @ Yelp (Enrico Canzonieri, 30m)
> -- How BNP Paribas Fortis uses Flink for real-time fraud detectionDavid
> Massart (David Massart, tentative)
>
> Please refer to the meetup page [1] for more details.
>
> Many thanks go to Yelp for their sponsorship. At the same time, we might
> still have room for one more short talk. Please let me know if interested.
>
>
> Thanks,
>
> Xuefu
>
> [1] https://www.meetup.com/Bay-Area-Apache-Flink-Meetup/events/262680261/
>
>

Re: Add Bucket File System Table Sink

[Jun Zhang]

Hi??Fabian ??


Thank you very much for your suggestion. This is when I use flink sql to write 
data to hdfs at work. I feel that it is inconvenient. I wrote this function, 
and then I want to contribute it to the community. This is my first PR , some 
processes may not be clear, I am very sorry.


Kurt suggested combining this feature with FLIP-63 because they have some 
common features, such as write data to file system with kinds of format, so I 
want to treat this function as a sub-task of FLIP-63. Add a partitionable 
bucket file system table sink.


I then added the document and sent a DISCUSS to explain my detailed design 
ideas and implementation. How do you see it?






-- Original --
From: Fabian Hueske https://flink.apache.org/contributing/contribute-code.html

Am Di., 17. Sept. 2019 um 04:39 Uhr schrieb Jun Zhang <825875...@qq.com:

 Hello everyone:
 I am a user and fan of flink. I also want to join the flink community. I
 contributed my first PR a few days ago. Can anyone help me to review my
 code? If there is something wrong, hope I would be grateful if you can give
 some advice.

 This PR is mainly in the process of development, I use sql to read data
 from kafka and then write to hdfs, I found that there is no suitable
 tablesink, I found the document and found that File System Connector is
 only experimental (
 
https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/connect.html#file-system-connector),
 so I wrote a Bucket File System Table Sink that supports writing stream
 data. Hdfs, file file system, data format supports json, csv, parquet,
 avro. Subsequently add other format support, such as protobuf, thrift, etc.

 In addition, I also added documentation, python api, units test,
 end-end-test, sql-client, DDL, and compiled on travis.

 the issue is https://issues.apache.org/jira/browse/FLINK-12584
 thank you very much

Re: [DISCUSS] FLIP-66: Support time attribute in SQL DDL

[Fabian Hueske]

Hi Jark,

Thanks for the summary!
I like the proposal!

It makes it very clear that an event time attribute is an existing column
on which watermark metadata is defined whereas a processing time attribute
is a computed field.

I have one comment regarding the section on "Complex Watermark Strategies".
The proposal says that you can also use a scalar function.
I don't think that a "text book" scalar function would be sufficient for
more advanced strategies.
For example a histogram-based approach would need to remember the values of
the last x records.
The interface of a scalar function would still work for that, but it would
be a stateful function (which would not be OK for a scalar function).
I don't think it's a problem, but wanted to mention it here.

Best, Fabian

Am Do., 19. Sept. 2019 um 18:05 Uhr schrieb Jark Wu :

> Hi everyone,
>
> Thanks all for the valuable suggestions and feedbacks so far.
> Before starting the vote, I would like to summarize the proposed DDL syntax
> in the mailing list.
>
> ## Rowtime Attribute (Watermark Syntax)
>
> CREATE TABLE table_name (
>   WATERMARK FOR  AS 
> ) WITH (
>   ...
> )
>
> It marks an existing field  as the rowtime attribute, and the
> watermark is generated by the expression .
>  can be arbitrary expression which returns a
> nullable BIGINT or TIMESTAMP as the watermark value.
>
> For common cases, users can use the following expressions to define a
> strategy.
> 1. Bounded Out of Orderness, the strategy can be "rowtimeField - INTERVAL
> 'string' timeUnit".
> 2. Preserve Watermark From Source, the strategy can be
> "SYSTEM_WATERMARK()".
>
> ## Proctime Attribute
>
> CREATE TABLE table_name (
>   ...
>   proc AS SYSTEM_PROCTIME()
> ) WITH (
>   ...
> )
>
> It uses the computed column syntax to add an additional column with
> proctime attribute. Here SYSTEM_PROCTIME() is a built-in function.
>
> For more details and the implementations, please refer to the design doc:
>
> https://docs.google.com/document/d/1-SecocBqzUh7zY6HBYcfMlG_0z-JAcuZkCvsmN3LrOw/edit?ts=5d822dba
>
> Feel free to leave your further feedbacks!
>
> Thanks,
> Jark
>
> On Thu, 19 Sep 2019 at 11:23, Kurt Young  wrote:
>
> > +1 to start vote process.
> >
> > Best,
> > Kurt
> >
> >
> > On Thu, Sep 19, 2019 at 10:54 AM Jark Wu  wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks all for joining the discussion in the doc[1].
> > > It seems that the discussion is converged and there is a consensus on
> the
> > > current FLIP document.
> > > If there is no objection, I would like to convert it into cwiki FLIP
> page
> > > and start voting process.
> > >
> > > For more details, please refer to the design doc (it is slightly
> changed
> > > since the initial proposal).
> > >
> > > Thanks,
> > > Jark
> > >
> > > [1]:
> > >
> > >
> >
> https://docs.google.com/document/d/1-SecocBqzUh7zY6HBYcfMlG_0z-JAcuZkCvsmN3LrOw/edit?ts=5d8258cd
> > >
> > > On Mon, 16 Sep 2019 at 16:12, Kurt Young  wrote:
> > >
> > > > After some review and discussion in the google document, I think it's
> > > time
> > > > to
> > > > convert this design to a cwiki flip page and start voting process.
> > > >
> > > > Best,
> > > > Kurt
> > > >
> > > >
> > > > On Mon, Sep 9, 2019 at 7:46 PM Jark Wu  wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Thanks all for so much feedbacks received in the doc so far.
> > > > > I saw a general agreement on using computed column to support
> > proctime
> > > > > attribute and extract timestamps.
> > > > > So we will prepare a computed column FLIP and share in the dev ML
> > soon.
> > > > >
> > > > > Feel free to leave more comments!
> > > > >
> > > > > Best,
> > > > > Jark
> > > > >
> > > > >
> > > > >
> > > > > On Fri, 6 Sep 2019 at 13:50, Dian Fu 
> wrote:
> > > > >
> > > > > > Hi Jark,
> > > > > >
> > > > > > Thanks for bringing up this discussion and the detailed design
> doc.
> > > > This
> > > > > > is definitely a critical feature for streaming SQL jobs. I have
> > left
> > > a
> > > > > few
> > > > > > comments in the design doc.
> > > > > >
> > > > > > Thanks,
> > > > > > Dian
> > > > > >
> > > > > > > 在 2019年9月6日，上午11:48，Forward Xu  写道：
> > > > > > >
> > > > > > > Thanks Jark for this topic, This will be very useful.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > ForwardXu
> > > > > > >
> > > > > > > Danny Chan  于2019年9月6日周五 上午11:26写道：
> > > > > > >
> > > > > > >> Thanks Jark for bring up this topic, this is definitely an
> > import
> > > > > > feature
> > > > > > >> for the SQL, especially the DDL users.
> > > > > > >>
> > > > > > >> I would spend some time to review this design doc, really
> > thanks.
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Danny Chan
> > > > > > >> 在 2019年9月6日 +0800 AM11:19，Jark Wu ，写道：
> > > > > > >>> Hi everyone,
> > > > > > >>>
> > > > > > >>> I would like to start discussion about how to support time
> > > > attribute
> > > > > in
> > > > > > >> SQL
> > > > > > >>> DDL.
> > > > > > >>> In Flink 1.9, we already introduced a 

[jira] [Created] (FLINK-14157) Temporarily remove S3 StreamingFileSink end-to-end test

[Kostas Kloudas (Jira)]

Kostas Kloudas created FLINK-14157:
--

 Summary: Temporarily remove S3 StreamingFileSink end-to-end test
 Key: FLINK-14157
 URL: https://issues.apache.org/jira/browse/FLINK-14157
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / FileSystem, Tests
Affects Versions: 1.10.0
Reporter: Kostas Kloudas
Assignee: Kostas Kloudas


This issue temporarily disables the failing test so that we can have a green 
travis build, until a proper solution is found.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)