[jira] [Created] (FLINK-8563) Support consecutive DOT operators

[Timo Walther (JIRA)]

Timo Walther created FLINK-8563:
---

 Summary: Support consecutive DOT operators 
 Key: FLINK-8563
 URL: https://issues.apache.org/jira/browse/FLINK-8563
 Project: Flink
  Issue Type: Improvement
  Components: Table API & SQL
Reporter: Timo Walther


We added support for accessing fields of arrays of composite types in 
FLINK-7923. However, accessing another nested subfield is not supported by 
Calcite. See CALCITE-2162. We should fix this once we upgrade to Calcite 1.16.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

[Bowen Li]

Hi guys,

I want to propose a new FLIP -- FLIP-25 - Support User State TTL Natively
in Flink. This has been one of most handy and most frequently asked
features in Flink community. The jira ticket is FLINK-3089
.

I've written a rough design

doc
,
and developed prototypes for both heap and rocksdb state backends.

My question is: shall we create a FLIP page for this? Can I be granted the
privileges of creating pages in
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
?

Thanks,
Bowen

[jira] [Created] (FLINK-8562) Fix YARNSessionFIFOSecuredITCase

[Shuyi Chen (JIRA)]

Shuyi Chen created FLINK-8562:
-

 Summary: Fix YARNSessionFIFOSecuredITCase
 Key: FLINK-8562
 URL: https://issues.apache.org/jira/browse/FLINK-8562
 Project: Flink
  Issue Type: Bug
  Components: Security
Reporter: Shuyi Chen
Assignee: Shuyi Chen


Currently, YARNSessionFIFOSecuredITCase will not fail even if the current Flink 
YARN Kerberos integration test is failing. Please see FLINK-8275.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Timestamp/watermark support in Kinesis consumer

[Thomas Weise]

Hi,

The Kinesis consumer currently does not emit watermarks, and this can lead
to problems when a single subtask reads from multiple shards and offsets
are not closely aligned with respect to the event time.

The Kafka consumer has support for periodic and punctuated watermarks,
although there is also the unresolved issue
https://issues.apache.org/jira/browse/FLINK-5479 that would equally apply
for Kinesis.

I propose adding support for timestamp assigner and watermark generator to
the Kinesis consumer.

As for handling of idle shards, is there a preference? Perhaps a
customization point on the assigner that defers the decision to the user
would be appropriate?

Thanks,
Thomas

Re: [DISCUSS] Releasing Flink 1.5.0

[Kostas Kloudas]

Hi Aljoscha,

I believe that support for Broadcast State should also be in 1.5.
There is an open PR https://github.com/apache/flink/pull/5230 
 for that
and there are some pending issues related to scala api and documentation.

Thanks,
Kostas

> On Feb 5, 2018, at 5:37 PM, Timo Walther  wrote:
> 
> Hi Shuyi,
> 
> I will take a look at it again this week. I'm pretty sure it will be part of 
> 1.5.0.
> 
> Regards,
> Timo
> 
> 
> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>> internal users waiting for this feature.
>> 
>> [FLINK-7923 ] Support
>> accessing subfields of a Composite element in an Object Array type column
>> 
>> Thanks a lot
>> Shuyi
>> 
>> 
>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif  wrote:
>> 
>>> Hi guys,
>>> 
>>> Sorry for jumping in, but I think
>>> 
>>> [FLINK-8101] Elasticsearch 6.X support
>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible with
>>> Elasticsearch 5.2+ client
>>> 
>>>  have long been awaited and there was one PR from me and from someone else
>>> showing the interest ;) So if you could consider it for 1.5 that would be
>>> great!
>>> 
>>> Thanks!
>>> --
>>> Christophe
>>> 
>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther  wrote:
>>> 
 Hi Aljoscha,
 
 it would be great if we can include the first version of the SQL client
 (see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
 we can merge this with explicit "experimental/alpha" status. It is far
>>> away
 from feature completeness but will be a great tool for Flink beginners.
 
 In order to use the SQL client we would need to also add some table
 sources with the new unified table factories (FLINK-8535), but this is
 optional because a user can implement own table factories at the
>>> begining.
 Regards,
 Timo
 
 
 Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
 
 Hi Aljoscha,
> Thanks for starting the discussion.
> 
> I think there’s a few connector related must-have improvements that we
> should get in before the feature freeze, since quite a few users have
>>> been
> asking for them:
> 
> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set
>>> up
> start offset
> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
> consider idle partitions
> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> FlinkKinesisConsumer
> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
> 
> These are still missing in the master branch. Only FLINK-5479 is still
> lacking a pull request.
> 
> Cheers,
> Gordon
> 
> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>> aljos...@apache.org)
> wrote:
> Hi Everyone,
> 
> When we decided to do the 1.4.0 release a while back we did that to get
>>> a
> stable release out before putting in a couple of new features. Back
>>> then,
> some of those new features (FLIP-6, network stack changes, local state
> recovery) were almost ready and we wanted to do a shortened 1.5.0
> development cycle to allow for those features to become ready and then
>>> do
> the next release.
> 
> We are now approaching the approximate time where we wanted to do the
> Flink 1.5.0 release so I would like to gauge where we are and gather
> opinions on how we should proceed now.
> 
> With this, I'd also like to propose myself as the release manager for
> 1.5.0 but I'm very happy to yield if someone else would be interested in
> doing that.
> 
> What do you think?
> 
> Best,
> Aljoscha
> 
 
 
>>> 
>>> --
>>> Christophe
>>> 
>> 
>> 
> 

Re: Fetch Metrics

[Amit Jain]

Hi Suvimal,

You may use REST API connecting to Job Manager to retrieve the stored
metrics. Have a look at this link https://ci.apache.org/
projects/flink/flink-docs-release-1.4/monitoring/metrics.html#rest-api-
integration

--
Thanks,
Amit

On Mon, Feb 5, 2018 at 2:47 PM, Suvimal Yashraj <
suvimal.yash...@ericsson.com> wrote:

> Hello,
> I'm new to apache flink and I want to extract the already present metrics
> in flink which is stored on the server of the currently running jobs like
> bytes transferred, cpu usage, duration, etc.
> I'm finding it difficult so as to how to connect my java program to the
> server and fetch these details.
> If there's any solution to this problem please provide.
>
> Thank You.
>
> Regards,
> Suvimal
>
>

Re: [DISCUSS] Releasing Flink 1.5.0

[Timo Walther]

Hi Shuyi,

I will take a look at it again this week. I'm pretty sure it will be 
part of 1.5.0.


Regards,
Timo


Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:

Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
internal users waiting for this feature.

[FLINK-7923 ] Support
accessing subfields of a Composite element in an Object Array type column

Thanks a lot
Shuyi


On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif  wrote:


Hi guys,

Sorry for jumping in, but I think

[FLINK-8101] Elasticsearch 6.X support
[FLINK-7386]  Flink Elasticsearch 5 connector is not compatible with
Elasticsearch 5.2+ client

  have long been awaited and there was one PR from me and from someone else
showing the interest ;) So if you could consider it for 1.5 that would be
great!

Thanks!
--
Christophe

On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther  wrote:


Hi Aljoscha,

it would be great if we can include the first version of the SQL client
(see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
we can merge this with explicit "experimental/alpha" status. It is far

away

from feature completeness but will be a great tool for Flink beginners.

In order to use the SQL client we would need to also add some table
sources with the new unified table factories (FLINK-8535), but this is
optional because a user can implement own table factories at the

begining.

Regards,
Timo


Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:

Hi Aljoscha,

Thanks for starting the discussion.

I think there’s a few connector related must-have improvements that we
should get in before the feature freeze, since quite a few users have

been

asking for them:

[FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set

up

start offset
[FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
consider idle partitions
[FLINK-8516] Pluggable shard-to-subtask partitioning for
FlinkKinesisConsumer
[FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer

These are still missing in the master branch. Only FLINK-5479 is still
lacking a pull request.

Cheers,
Gordon

On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (

aljos...@apache.org)

wrote:
Hi Everyone,

When we decided to do the 1.4.0 release a while back we did that to get

a

stable release out before putting in a couple of new features. Back

then,

some of those new features (FLIP-6, network stack changes, local state
recovery) were almost ready and we wanted to do a shortened 1.5.0
development cycle to allow for those features to become ready and then

do

the next release.

We are now approaching the approximate time where we wanted to do the
Flink 1.5.0 release so I would like to gauge where we are and gather
opinions on how we should proceed now.

With this, I'd also like to propose myself as the release manager for
1.5.0 but I'm very happy to yield if someone else would be interested in
doing that.

What do you think?

Best,
Aljoscha






--
Christophe

Re: [DISCUSS] Releasing Flink 1.5.0

[Shuyi Chen]

Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
internal users waiting for this feature.

[FLINK-7923 ] Support
accessing subfields of a Composite element in an Object Array type column

Thanks a lot
Shuyi


On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif  wrote:

> Hi guys,
>
> Sorry for jumping in, but I think
>
> [FLINK-8101] Elasticsearch 6.X support
> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible with
> Elasticsearch 5.2+ client
>
>  have long been awaited and there was one PR from me and from someone else
> showing the interest ;) So if you could consider it for 1.5 that would be
> great!
>
> Thanks!
> --
> Christophe
>
> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther  wrote:
>
> > Hi Aljoscha,
> >
> > it would be great if we can include the first version of the SQL client
> > (see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
> > we can merge this with explicit "experimental/alpha" status. It is far
> away
> > from feature completeness but will be a great tool for Flink beginners.
> >
> > In order to use the SQL client we would need to also add some table
> > sources with the new unified table factories (FLINK-8535), but this is
> > optional because a user can implement own table factories at the
> begining.
> >
> > Regards,
> > Timo
> >
> >
> > Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> >
> > Hi Aljoscha,
> >>
> >> Thanks for starting the discussion.
> >>
> >> I think there’s a few connector related must-have improvements that we
> >> should get in before the feature freeze, since quite a few users have
> been
> >> asking for them:
> >>
> >> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set
> up
> >> start offset
> >> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
> >> consider idle partitions
> >> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> >> FlinkKinesisConsumer
> >> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
> >>
> >> These are still missing in the master branch. Only FLINK-5479 is still
> >> lacking a pull request.
> >>
> >> Cheers,
> >> Gordon
> >>
> >> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
> aljos...@apache.org)
> >> wrote:
> >> Hi Everyone,
> >>
> >> When we decided to do the 1.4.0 release a while back we did that to get
> a
> >> stable release out before putting in a couple of new features. Back
> then,
> >> some of those new features (FLIP-6, network stack changes, local state
> >> recovery) were almost ready and we wanted to do a shortened 1.5.0
> >> development cycle to allow for those features to become ready and then
> do
> >> the next release.
> >>
> >> We are now approaching the approximate time where we wanted to do the
> >> Flink 1.5.0 release so I would like to gauge where we are and gather
> >> opinions on how we should proceed now.
> >>
> >> With this, I'd also like to propose myself as the release manager for
> >> 1.5.0 but I'm very happy to yield if someone else would be interested in
> >> doing that.
> >>
> >> What do you think?
> >>
> >> Best,
> >> Aljoscha
> >>
> >
> >
> >
>
>
> --
> Christophe
>



-- 
"So you have to trust that the dots will somehow connect in your future."

Re: [DISCUSS] Releasing Flink 1.5.0

[Christophe Jolif]

Hi guys,

Sorry for jumping in, but I think

[FLINK-8101] Elasticsearch 6.X support
[FLINK-7386]  Flink Elasticsearch 5 connector is not compatible with
Elasticsearch 5.2+ client

 have long been awaited and there was one PR from me and from someone else
showing the interest ;) So if you could consider it for 1.5 that would be
great!

Thanks!
--
Christophe

On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther  wrote:

> Hi Aljoscha,
>
> it would be great if we can include the first version of the SQL client
> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
> we can merge this with explicit "experimental/alpha" status. It is far away
> from feature completeness but will be a great tool for Flink beginners.
>
> In order to use the SQL client we would need to also add some table
> sources with the new unified table factories (FLINK-8535), but this is
> optional because a user can implement own table factories at the begining.
>
> Regards,
> Timo
>
>
> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>
> Hi Aljoscha,
>>
>> Thanks for starting the discussion.
>>
>> I think there’s a few connector related must-have improvements that we
>> should get in before the feature freeze, since quite a few users have been
>> asking for them:
>>
>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set up
>> start offset
>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>> consider idle partitions
>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>> FlinkKinesisConsumer
>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>
>> These are still missing in the master branch. Only FLINK-5479 is still
>> lacking a pull request.
>>
>> Cheers,
>> Gordon
>>
>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (aljos...@apache.org)
>> wrote:
>> Hi Everyone,
>>
>> When we decided to do the 1.4.0 release a while back we did that to get a
>> stable release out before putting in a couple of new features. Back then,
>> some of those new features (FLIP-6, network stack changes, local state
>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>> development cycle to allow for those features to become ready and then do
>> the next release.
>>
>> We are now approaching the approximate time where we wanted to do the
>> Flink 1.5.0 release so I would like to gauge where we are and gather
>> opinions on how we should proceed now.
>>
>> With this, I'd also like to propose myself as the release manager for
>> 1.5.0 but I'm very happy to yield if someone else would be interested in
>> doing that.
>>
>> What do you think?
>>
>> Best,
>> Aljoscha
>>
>
>
>


-- 
Christophe

Re: [DISCUSS] Releasing Flink 1.5.0

[Timo Walther]

Hi Aljoscha,

it would be great if we can include the first version of the SQL client 
(see FLIP-24, Implementation Plan 1). I will open a PR this week. I 
think we can merge this with explicit "experimental/alpha" status. It is 
far away from feature completeness but will be a great tool for Flink 
beginners.


In order to use the SQL client we would need to also add some table 
sources with the new unified table factories (FLINK-8535), but this is 
optional because a user can implement own table factories at the begining.


Regards,
Timo


Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:

Hi Aljoscha,

Thanks for starting the discussion.

I think there’s a few connector related must-have improvements that we should 
get in before the feature freeze, since quite a few users have been asking for 
them:

[FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set up start 
offset
[FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should consider 
idle partitions
[FLINK-8516] Pluggable shard-to-subtask partitioning for FlinkKinesisConsumer
[FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer

These are still missing in the master branch. Only FLINK-5479 is still lacking 
a pull request.

Cheers,
Gordon

On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (aljos...@apache.org) wrote:
Hi Everyone,

When we decided to do the 1.4.0 release a while back we did that to get a 
stable release out before putting in a couple of new features. Back then, some 
of those new features (FLIP-6, network stack changes, local state recovery) 
were almost ready and we wanted to do a shortened 1.5.0 development cycle to 
allow for those features to become ready and then do the next release.

We are now approaching the approximate time where we wanted to do the Flink 
1.5.0 release so I would like to gauge where we are and gather opinions on how 
we should proceed now.

With this, I'd also like to propose myself as the release manager for 1.5.0 but 
I'm very happy to yield if someone else would be interested in doing that.

What do you think?

Best,
Aljoscha

Re: [DISCUSS] Releasing Flink 1.5.0

[Tzu-Li (Gordon) Tai]

Hi Aljoscha,

Thanks for starting the discussion.

I think there’s a few connector related must-have improvements that we should 
get in before the feature freeze, since quite a few users have been asking for 
them:

[FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set up start 
offset
[FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should consider 
idle partitions
[FLINK-8516] Pluggable shard-to-subtask partitioning for FlinkKinesisConsumer
[FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer

These are still missing in the master branch. Only FLINK-5479 is still lacking 
a pull request.

Cheers,
Gordon

On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (aljos...@apache.org) wrote:
Hi Everyone,  

When we decided to do the 1.4.0 release a while back we did that to get a 
stable release out before putting in a couple of new features. Back then, some 
of those new features (FLIP-6, network stack changes, local state recovery) 
were almost ready and we wanted to do a shortened 1.5.0 development cycle to 
allow for those features to become ready and then do the next release.  

We are now approaching the approximate time where we wanted to do the Flink 
1.5.0 release so I would like to gauge where we are and gather opinions on how 
we should proceed now.  

With this, I'd also like to propose myself as the release manager for 1.5.0 but 
I'm very happy to yield if someone else would be interested in doing that.  

What do you think?  

Best,  
Aljoscha

[jira] [Created] (FLINK-8561) SharedBuffer line 573 uses == to compare BufferEntries instead of .equals.

[Kostas Kloudas (JIRA)]

Kostas Kloudas created FLINK-8561:
-

 Summary: SharedBuffer line 573 uses == to compare BufferEntries 
instead of .equals.
 Key: FLINK-8561
 URL: https://issues.apache.org/jira/browse/FLINK-8561
 Project: Flink
  Issue Type: Bug
  Components: CEP
Affects Versions: 1.4.0
Reporter: Kostas Kloudas
Assignee: Kostas Kloudas
 Fix For: 1.4.1






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8560) Access to the current key in ProcessFunction after keyBy()

[JIRA]

Jürgen Thomann created FLINK-8560:
-

 Summary: Access to the current key in ProcessFunction after keyBy()
 Key: FLINK-8560
 URL: https://issues.apache.org/jira/browse/FLINK-8560
 Project: Flink
  Issue Type: Wish
  Components: DataStream API
Reporter: Jürgen Thomann


Currently it is required to store the key of a keyBy() in the processElement 
method to have access to it in the OnTimerContext.

This is not so good as you have to check in the processElement method for every 
element if the key is already stored and set it if it's not already set.

A possible solution would adding OnTimerContext#getCurrentKey() or a similar 
method. Maybe having it in the open() method could maybe work as well.

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Getting-Key-from-keyBy-in-ProcessFunction-tt18126.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] FLIP-23 Model Serving

[Stavros Kontopoulos]

Thanx @Fabian. I will update the document accordingly wrt metrics.
I agree there are pros and cons.

Best,
Stavros


On Wed, Jan 31, 2018 at 1:07 AM, Fabian Hueske  wrote:

> OK, I think there was plenty of time to comment on this FLIP.
> I'll move it to the ACCEPTED status.
>
> @Stavros, please consider the feedback regarding the metrics.
> I agree with Chesnay that metrics should be primarily exposed via the
> metrics system.
> Storing them in state makes them fault-tolerant and queryable if the state
> is properly configured.
>
> Thanks,
> Fabian
>
> 2018-01-22 17:19 GMT+01:00 Chesnay Schepler :
>
> > I'm currently looking over it, but one thing that stood out was that the
> > FLIP proposes to use queryable state
> > as a monitoring solution. Given that we have a metric system that
> > integrates with plenty of commonly used
> > metric backends this doesn't really make sense to me.
> >
> > Storing them in state still has value in terms of fault-tolerance though,
> > since this is something that the metric
> > system doesn't provide by itself.
> >
> >
> > On 18.01.2018 13:57, Fabian Hueske wrote:
> >
> >> Are there any more comments on the FLIP?
> >>
> >> Otherwise, I'd suggest to move the FLIP to the accepted FLIPs [1] and
> >> continue with the implementation.
> >>
> >> Also, is there a committer who'd like to shepherd the FLIP and review
> the
> >> corresponding PRs?
> >> Of course, everybody is welcome to review the code but we need at least
> >> one
> >> committer who will eventually merge the changes.
> >>
> >> Best,
> >> Fabian
> >>
> >> [1]
> >> https://cwiki.apache.org/confluence/display/FLINK/Flink+
> >> Improvement+Proposals
> >>
> >> 2017-12-04 10:54 GMT+01:00 Fabian Hueske :
> >>
> >> Hi,
> >>>
> >>> Sorry for the late follow up.
> >>>
> >>> I think I understand the motivation for choosing ProtoBuf as the
> >>> representation and serialization format and this makes sense to me.
> >>>
> >>> However, it might be a good idea to provide tooling to convert Flink
> >>> types
> >>> (described as TypeInformation) to ProtoBuf.
> >>> Otherwise, users of the model serving library would need to manually
> >>> convert their data types (say Scala tuples, case classes, or Avro
> Pojos)
> >>> to
> >>> ProtoBuf messages.
> >>> I don't think that this needs to be included in the first version but
> it
> >>> might be a good extension to make the library easier to use.
> >>>
> >>> Best,
> >>> Fabian
> >>>
> >>>
> >>>
> >>> 2017-11-28 17:22 GMT+01:00 Boris Lublinsky <
> >>> boris.lublin...@lightbend.com>
> >>> :
> >>>
> >>> Thanks Fabian,
>  More below
> 
> 
> 
>  Boris Lublinsky
>  FDP Architect
>  boris.lublin...@lightbend.com
>  https://www.lightbend.com/
> 
>  On Nov 28, 2017, at 8:21 AM, Fabian Hueske  wrote:
> 
>  Hi Boris and Stavros,
> 
>  Thanks for the responses.
> 
>  Ad 1) Thanks for the clarification. I think I misunderstood this part
> of
>  the proposal.
>  I interpreted the argument why to chose ProtoBuf for network encoding
>  ("ability
>  to represent different data types") such that different a model
> pipeline
>  should work on different data types.
>  I agree that it should be possible to give records of the same type
> (but
>  with different keys) to different models. The key-based join approach
>  looks
>  good to me.
> 
>  Ad 2) I understand that ProtoBuf is a good choice to serialize models
>  for
>  the given reasons.
>  However, the choice of ProtoBuf serialization for the records might
> make
>  the integration with existing libraries and also regular DataStream
>  programs more difficult.
>  They all use Flink's TypeSerializer system to serialize and
> deserialize
>  records by default. Hence, we would need to add a conversion step
> before
>  records can be passed to a model serving operator.
>  Are you expecting some common format that all records follow (such as
> a
>  Row or Vector type) or do you plan to support arbitrary records such
> as
>  Pojos?
>  If you plan for a specific type, you could add a TypeInformation for
>  this
>  type with a TypeSerializer that is based on ProtoBuf.
> 
>  The way I look at it is slightly different. The common format for
>  records, supported by Flink, is Byte array with a little bit of
> header,
>  describing data type and is used for routing. The actual unmarshalling
>  is
>  done by the model implementation itself. This provides the maximum
>  flexibility and gives user the freedom to create his own types without
>  breaking underlying framework.
> 
>  Ad 4) @Boris: I made this point not about the serialization format but
>  how the library would integrate with Flink's DataStream API.
>  I thought I had seen a code snippet that showed a new method on the
>  DataStream object but cannot find this anymore.
>  So, I just wanted t

[jira] [Created] (FLINK-8559) JobManagerHACheckpointRecoveryITCase runs indefinitely on Windows

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-8559:
---

 Summary: JobManagerHACheckpointRecoveryITCase runs indefinitely on 
Windows
 Key: FLINK-8559
 URL: https://issues.apache.org/jira/browse/FLINK-8559
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing, Tests
Affects Versions: 1.5.0
Reporter: Chesnay Schepler


The 
{color:#33}{{testCheckpointedStreamingProgramIncrementalRocksDB}}{color} 
test in {color:#33}{{JobManagerHACheckpointRecoveryITCase}}{color} runs 
indefinitely on Windows.

 

The snapshotting fails for one of 2 tasks due to FLINK-8557, but the job never 
enters a failure state. For subsequent checkpoints the failed task is 
completely ignored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8558) Add unified format interfaces and format discovery

[Timo Walther (JIRA)]

Timo Walther created FLINK-8558:
---

 Summary: Add unified format interfaces and format discovery
 Key: FLINK-8558
 URL: https://issues.apache.org/jira/browse/FLINK-8558
 Project: Flink
  Issue Type: New Feature
  Components: Streaming Connectors
Reporter: Timo Walther
Assignee: Timo Walther


In the last release, we introduced a new module {{flink-formats}}. Currently 
only {{flink-avro}} is located there but we will add more formats such as 
{{flink-json}}, {{flink-protobuf}}, and so on. For better separation of 
concerns we want decouple connectors from formats: e.g., remove 
{{KafkaAvroTableSource}} and {{KafkaJsonTableSource}}.

A newly introduced {{FormatFactory}} will use Java service loaders to discovery 
available formats in the classpath (similar to how file systems are discovered 
now). A {{Format}} will provide a method for converting {{byte[]}} to target 
record type.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8557) OperatorSubtaskDescriptionText causes failures on Windows

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-8557:
---

 Summary: OperatorSubtaskDescriptionText causes failures on Windows
 Key: FLINK-8557
 URL: https://issues.apache.org/jira/browse/FLINK-8557
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing, Streaming
Affects Versions: 1.5.0
Reporter: Chesnay Schepler


File-based state backends that use the description provided by 
{{OperatorSubtaskDescriptionText}} will categorically fail on Windows as they 
contain characters that aren't allowed in paths ({{(,) and /).}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Fetch Metrics

[Suvimal Yashraj]

Hello,
I'm new to apache flink and I want to extract the already present metrics in 
flink which is stored on the server of the currently running jobs like bytes 
transferred, cpu usage, duration, etc.
I'm finding it difficult so as to how to connect my java program to the server 
and fetch these details.
If there's any solution to this problem please provide.

Thank You.

Regards,
Suvimal