Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

[Richard Deurwaarder]

Hello all,

I noticed the PubSub example jar is not included in the examples/ dir of
flink-dist. I've created https://issues.apache.org/jira/browse/FLINK-13700
 + https://github.com/apache/flink/pull/9424/files to fix this.

I will leave it up to you to decide if we want to add this to 1.9.0.

Regards,

Richard

On Tue, Aug 13, 2019 at 9:04 AM Till Rohrmann  wrote:

> Hi Jark,
>
> thanks for reporting this issue. Could this be a documented limitation of
> Blink's preview version? I think we have agreed that the Blink SQL planner
> will be rather a preview feature than production ready. Hence it could
> still contain some bugs. My concern is that there might be still other
> issues which we'll discover bit by bit and could postpone the release even
> further if we say Blink bugs are blockers.
>
> Cheers,
> Till
>
> On Tue, Aug 13, 2019 at 7:42 AM Jark Wu  wrote:
>
> > Hi all,
> >
> > I just find an issue when testing connector DDLs against blink planner
> for
> > rc2.
> > This issue lead to the DDL doesn't work when containing
> timestamp/date/time
> > type.
> > I have created an issue FLINK-13699[1] and a pull request for this.
> >
> > IMO, this can be a blocker issue of 1.9 release. Because
> > timestamp/date/time are primitive types, and this will break the DDL
> > feature.
> > However, I want to hear more thoughts from the community whether we
> should
> > recognize it as a blocker.
> >
> > Thanks,
> > Jark
> >
> >
> > [1]: https://issues.apache.org/jira/browse/FLINK-13699
> >
> >
> >
> > On Mon, 12 Aug 2019 at 22:46, Becket Qin  wrote:
> >
> > > Thanks Gordon, will do that.
> > >
> > > On Mon, Aug 12, 2019 at 4:42 PM Tzu-Li (Gordon) Tai <
> tzuli...@apache.org
> > >
> > > wrote:
> > >
> > > > Concerning FLINK-13231:
> > > >
> > > > Since this is a @PublicEvolving interface, technically it is ok to
> > break
> > > > it across releases (including across bugfix releases?).
> > > > So, @Becket if you do merge it now, please mark the fix version as
> > 1.9.1.
> > > >
> > > > During the voting process, in the case a new RC is created, we
> usually
> > > > check the list of changes compared to the previous RC, and correct
> the
> > > "Fix
> > > > Version" of the corresponding JIRAs to be the right version (in the
> > case,
> > > > it would be corrected to 1.9.0 instead of 1.9.1).
> > > >
> > > > On Mon, Aug 12, 2019 at 4:25 PM Till Rohrmann 
> > > > wrote:
> > > >
> > > >> I agree that it would be nicer. Not sure whether we should cancel
> the
> > RC
> > > >> for this issue given that it is open for quite some time and hasn't
> > been
> > > >> addressed until very recently. Maybe we could include it on the
> > > shortlist
> > > >> of nice-to-do things which we do in case that the RC gets cancelled.
> > > >>
> > > >> Cheers,
> > > >> Till
> > > >>
> > > >> On Mon, Aug 12, 2019 at 4:18 PM Becket Qin 
> > > wrote:
> > > >>
> > > >>> Hi Till,
> > > >>>
> > > >>> Yes, I think we have already documented in that way. So technically
> > > >>> speaking it is fine to change it later. It is just better if we
> could
> > > >>> avoid
> > > >>> doing that.
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Jiangjie (Becket) Qin
> > > >>>
> > > >>> On Mon, Aug 12, 2019 at 4:09 PM Till Rohrmann <
> trohrm...@apache.org>
> > > >>> wrote:
> > > >>>
> > > >>> > Could we say that the PubSub connector is public evolving
> instead?
> > > >>> >
> > > >>> > Cheers,
> > > >>> > Till
> > > >>> >
> > > >>> > On Mon, Aug 12, 2019 at 3:18 PM Becket Qin  >
> > > >>> wrote:
> > > >>> >
> > > >>> > > Hi all,
> > > >>> > >
> > > >>> > > FLINK-13231(palindrome!) has a minor Google PubSub connector
> API
> > > >>> change
> > > >>> > > regarding how to config rate limiting. The GCP PubSub connector
> > is
> > > a
> > > >>> > newly
> > > >>> > > introduced connector in 1.9, so it would be nice to include
> this
> > > >>> change
> > > >>> > > into 1.9 rather than later to avoid a public API change. I am
> > > >>> thinking of
> > > >>> > > making this as a blocker for 1.9. Want to check what do others
> > > think.
> > > >>> > >
> > > >>> > > Thanks,
> > > >>> > >
> > > >>> > > Jiangjie (Becket) Qin
> > > >>> > >
> > > >>> > > On Mon, Aug 12, 2019 at 2:04 PM Zili Chen <
> wander4...@gmail.com>
> > > >>> wrote:
> > > >>> > >
> > > >>> > > > Hi Kurt,
> > > >>> > > >
> > > >>> > > > Thanks for your explanation. For [1] I think at least we
> should
> > > >>> change
> > > >>> > > > the JIRA issue field, like unset the fixed version. For [2] I
> > can
> > > >>> see
> > > >>> > > > the change is all in test scope but wonder if such a commit
> > still
> > > >>> > invalid
> > > >>> > > > the release candidate. IIRC previous RC VOTE threads would
> > > contain
> > > >>> a
> > > >>> > > > release manual/guide, I will try to look up it, too.
> > > >>> > > >
> > > >>> > > > Best,
> > > >>> > > > tison.
> > > >>> > > >
> > > >>> > > >
> > > >>> > > > Kurt Young  于2019年8月12日周一 下午5:42写道：
> > > >>> > > >
> > > >>> > > > > Hi Zili,
> > > >>> > > > >
> > > >>> > > > > 

[jira] [Created] (FLINK-13700) PubSub connector example not included in flink-dist

[Richard Deurwaarder (JIRA)]

Richard Deurwaarder created FLINK-13700:
---

 Summary: PubSub connector example not included in flink-dist
 Key: FLINK-13700
 URL: https://issues.apache.org/jira/browse/FLINK-13700
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Google Cloud PubSub
Affects Versions: 1.9.0
Reporter: Richard Deurwaarder






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [ANNOUNCE] Jiangjie (Becket) Qin has been added as a committer to the Flink project

[Richard Deurwaarder]

Congrats Becket! :)

Richard

On Thu, Jul 18, 2019 at 5:52 PM Xuefu Z  wrote:

> Congratulation, Becket! At least you're able to assign JIRAs now!
>
> On Thu, Jul 18, 2019 at 8:22 AM Rong Rong  wrote:
>
> > Congratulations Becket!
> >
> > --
> > Rong
> >
> > On Thu, Jul 18, 2019 at 7:05 AM Xingcan Cui  wrote:
> >
> > > Congrats Becket!
> > >
> > > Best,
> > > Xingcan
> > >
> > > On Thu, Jul 18, 2019, 07:17 Dian Fu  wrote:
> > >
> > > > Congrats Becket!
> > > >
> > > > > 在 2019年7月18日，下午6:42，Danny Chan  写道：
> > > > >
> > > > >> Congratulations!
> > > > >
> > > > > Best,
> > > > > Danny Chan
> > > > > 在 2019年7月18日 +0800 PM6:29，Haibo Sun ，写道：
> > > > >> Congratulations Becket!Best,
> > > > >> Haibo
> > > > >> 在 2019-07-18 17:51:06，"Hequn Cheng"  写道：
> > > > >>> Congratulations Becket!
> > > > >>>
> > > > >>> Best, Hequn
> > > > >>>
> > > > >>> On Thu, Jul 18, 2019 at 5:34 PM vino yang  >
> > > > wrote:
> > > > >>>
> > > >  Congratulations!
> > > > 
> > > >  Best,
> > > >  Vino
> > > > 
> > > >  Yun Gao  于2019年7月18日周四 下午5:31写道：
> > > > 
> > > > > Congratulations!
> > > > >
> > > > > Best,
> > > > > Yun
> > > > >
> > > > >
> > > > >
> > --
> > > > > From:Kostas Kloudas 
> > > > > Send Time:2019 Jul. 18 (Thu.) 17:30
> > > > > To:dev 
> > > > > Subject:Re: [ANNOUNCE] Jiangjie (Becket) Qin has been added as
> a
> > > >  committer
> > > > > to the Flink project
> > > > >
> > > > > Congratulations Becket!
> > > > >
> > > > > Kostas
> > > > >
> > > > > On Thu, Jul 18, 2019 at 11:21 AM Guowei Ma <
> guowei@gmail.com
> > >
> > > > wrote:
> > > > >
> > > > >> Congrats Becket!
> > > > >>
> > > > >> Best,
> > > > >> Guowei
> > > > >>
> > > > >>
> > > > >> Terry Wang  于2019年7月18日周四 下午5:17写道：
> > > > >>
> > > > >>> Congratulations Becket!
> > > > >>>
> > > >  在 2019年7月18日，下午5:09，Dawid Wysakowicz <
> dwysakow...@apache.org>
> > > 写道：
> > > > 
> > > >  Congratulations Becket! Good to have you onboard!
> > > > 
> > > >  On 18/07/2019 10:56, Till Rohrmann wrote:
> > > > > Congrats Becket!
> > > > >
> > > > > On Thu, Jul 18, 2019 at 10:52 AM Jeff Zhang <
> > zjf...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Congratulations Becket!
> > > > >>
> > > > >> Xu Forward  于2019年7月18日周四
> 下午4:39写道：
> > > > >>
> > > > >>> Congratulations Becket! Well deserved.
> > > > >>>
> > > > >>>
> > > > >>> Cheers,
> > > > >>>
> > > > >>> forward
> > > > >>>
> > > > >>> Kurt Young  于2019年7月18日周四 下午4:20写道：
> > > > >>>
> > > >  Congrats Becket!
> > > > 
> > > >  Best,
> > > >  Kurt
> > > > 
> > > > 
> > > >  On Thu, Jul 18, 2019 at 4:12 PM JingsongLee <
> > > > >> lzljs3620...@aliyun.com
> > > >  .invalid>
> > > >  wrote:
> > > > 
> > > > > Congratulations Becket!
> > > > >
> > > > > Best, Jingsong Lee
> > > > >
> > > > >
> > > > >
> > > > >
> > --
> > > > > From:Congxian Qiu 
> > > > > Send Time:2019年7月18日(星期四) 16:09
> > > > > To:dev@flink.apache.org 
> > > > > Subject:Re: [ANNOUNCE] Jiangjie (Becket) Qin has been
> > added
> > > >  as a
> > > >  committer
> > > > > to the Flink project
> > > > >
> > > > > Congratulations Becket! Well deserved.
> > > > >
> > > > > Best,
> > > > > Congxian
> > > > >
> > > > >
> > > > > Jark Wu  于2019年7月18日周四 下午4:03写道：
> > > > >
> > > > >> Congratulations Becket! Well deserved.
> > > > >>
> > > > >> Cheers,
> > > > >> Jark
> > > > >>
> > > > >> On Thu, 18 Jul 2019 at 15:56, Paul Lam <
> > > >  paullin3...@gmail.com>
> > > > >>> wrote:
> > > > >>> Congrats Becket!
> > > > >>>
> > > > >>> Best,
> > > > >>> Paul Lam
> > > > >>>
> > > >  在 2019年7月18日，15:41，Robert Metzger <
> > rmetz...@apache.org>
> > > >  写道：
> > > > 
> > > >  Hi all,
> > > > 
> > > >  I'm excited to announce that Jiangjie (Becket) Qin
> > just
> > > > > became
> > > > >> a
> > > > > Flink
> > > >  committer!
> > > > 
> > > >  Congratulations Becket!
> > > > 
> > > >  Best,
> > > > 

[jira] [Created] (FLINK-13231) Add a ratelimiter to pubsub source

[Richard Deurwaarder (JIRA)]

Richard Deurwaarder created FLINK-13231:
---

 Summary: Add a ratelimiter to pubsub source
 Key: FLINK-13231
 URL: https://issues.apache.org/jira/browse/FLINK-13231
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Google Cloud PubSub
Reporter: Richard Deurwaarder


Replace MaxMessagesToAcknowledge limit by introducing a rate limiter. See: 
[https://github.com/apache/flink/pull/6594#discussion_r300215868]



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13230) Retry acknowledgement calls

[Richard Deurwaarder (JIRA)]

Richard Deurwaarder created FLINK-13230:
---

 Summary: Retry acknowledgement calls
 Key: FLINK-13230
 URL: https://issues.apache.org/jira/browse/FLINK-13230
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Google Cloud PubSub
Reporter: Richard Deurwaarder


Currently when a pubsub/grpc pull fails we retry based on configuration given 
by the user.

We should do the same for  Acknowledgement calls



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13083) Various improvements PubSub Connector

[Richard Deurwaarder (JIRA)]

Richard Deurwaarder created FLINK-13083:
---

 Summary: Various improvements PubSub Connector
 Key: FLINK-13083
 URL: https://issues.apache.org/jira/browse/FLINK-13083
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Google Cloud PubSub
Affects Versions: 1.9.0
Reporter: Richard Deurwaarder


* Add retry mechanism to Acknowledgement calls



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Key state does not support migration

[Richard Deurwaarder]

Hello,

I am running into the problem where (avro) schema evolution works perfectly
for operator/keyed state but does not work when used with keyBy().

For example:

I have a job like so:

env.addSource(someSource())
  .keyBy(object -> getMyAvroObject())
  .process(doSomething())
  .addSink(someSink());

Where MyAvroObject has the following avdl:

enum SomeEnum{
  A,B
}

record SomeKey {
  SomeEnum someEnum;
}

This works fine but when I change my avro object to:


enum SomeEnum{
  A,B,C
}

record SomeKey {
  SomeEnum someEnum;
}


So with the added "C" in SomeEnum. If I restart my job (from savepoint)
with this new schema I get the following exception:

Caused by: org.apache.flink.util.StateMigrationException: The new key
serializer must be compatible.


Coming from this piece of code
(https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapRestoreOperation.java#L141):


if (keySerializerSchemaCompat.isCompatibleAfterMigration() ||
keySerializerSchemaCompat.isIncompatible()) {
throw new StateMigrationException("The new key serializer must be
compatible.");
}



My question is:


What is the reason key serializer / key state explicitly does not
support state migration? And is there any way to work around this?


Regards,


Richard

[jira] [Created] (FLINK-12325) Statsd reporter gives wrong metrics when using negative numbers

[Richard Deurwaarder (JIRA)]

Richard Deurwaarder created FLINK-12325:
---

 Summary: Statsd reporter gives wrong metrics when using negative 
numbers
 Key: FLINK-12325
 URL: https://issues.apache.org/jira/browse/FLINK-12325
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Metrics
Affects Versions: 1.8.0, 1.7.2, 1.6.4
Reporter: Richard Deurwaarder


The statsd reporter has a bug I believe when using negative numbers.

 

This is because when a metric is sent it is first converted to a string value. 
This means 100 becomes "100" and -100 becomes "-100". This value is then sent 
to statsd as a gauge value.

See this line for the conversion to string: 
[https://github.com/apache/flink/blob/master/flink-metrics/flink-metrics-statsd/src/main/java/org/apache/flink/metrics/statsd/StatsDReporter.java#L130]

And this line for sending it as a gauge:

[https://github.com/apache/flink/blob/master/flink-metrics/flink-metrics-statsd/src/main/java/org/apache/flink/metrics/statsd/StatsDReporter.java#L184]

 

So a value of -100 will be sent like this:
{code:java}
:-100|g{code}
 

The statsd protocol how ever states the following 
([https://github.com/statsd/statsd/blob/master/docs/metric_types.md#gauges]):
{noformat}
Adding a sign to the gauge value will change the value, rather than setting it.
{noformat}
 

 

So sending -100 multiple times means the gauge in statsd will be decremented 
multiple times, rather than set to -100.

 

I believe this isn't how flink expects it to work, isn't it?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] Apache Flink 1.8.0 released

[Richard Deurwaarder]

Very nice! Thanks Aljoscha and all contributors!

I have one question, will the docker image for 1.8.0 be released soon as
well? https://hub.docker.com/_/flink has the versions up to 1.7.2.

Regards,

Richard

On Wed, Apr 10, 2019 at 4:54 PM Rong Rong  wrote:

> Congrats! Thanks Aljoscha for being the release manager and all for making
> the release possible.
>
> --
> Rong
>
>
> On Wed, Apr 10, 2019 at 4:23 AM Stefan Richter 
> wrote:
>
>> Congrats and thanks to Aljoscha for managing the release!
>>
>> Best,
>> Stefan
>>
>> > On 10. Apr 2019, at 13:01, Biao Liu  wrote:
>> >
>> > Great news! Thanks Aljoscha and all the contributors.
>> >
>> > Till Rohrmann mailto:trohrm...@apache.org>>
>> 于2019年4月10日周三 下午6:11写道：
>> > Thanks a lot to Aljoscha for being our release manager and to the
>> community making this release possible!
>> >
>> > Cheers,
>> > Till
>> >
>> > On Wed, Apr 10, 2019 at 12:09 PM Hequn Cheng > > wrote:
>> > Thanks a lot for the great release Aljoscha!
>> > Also thanks for the work by the whole community. :-)
>> >
>> > Best, Hequn
>> >
>> > On Wed, Apr 10, 2019 at 6:03 PM Fabian Hueske > > wrote:
>> > Congrats to everyone!
>> >
>> > Thanks Aljoscha and all contributors.
>> >
>> > Cheers, Fabian
>> >
>> > Am Mi., 10. Apr. 2019 um 11:54 Uhr schrieb Congxian Qiu <
>> qcx978132...@gmail.com >:
>> > Cool!
>> >
>> > Thanks Aljoscha a lot for being our release manager, and all the others
>> who make this release possible.
>> >
>> > Best, Congxian
>> > On Apr 10, 2019, 17:47 +0800, Jark Wu > imj...@gmail.com>>, wrote:
>> > > Cheers!
>> > >
>> > > Thanks Aljoscha and all others who make 1.8.0 possible.
>> > >
>> > > On Wed, 10 Apr 2019 at 17:33, vino yang > > wrote:
>> > >
>> > > > Great news!
>> > > >
>> > > > Thanks Aljoscha for being the release manager and thanks to all the
>> > > > contributors!
>> > > >
>> > > > Best,
>> > > > Vino
>> > > >
>> > > > Driesprong, Fokko  于2019年4月10日周三 下午4:54写道：
>> > > >
>> > > > > Great news! Great effort by the community to make this happen.
>> Thanks all!
>> > > > >
>> > > > > Cheers, Fokko
>> > > > >
>> > > > > Op wo 10 apr. 2019 om 10:50 schreef Shaoxuan Wang <
>> wshaox...@gmail.com >:
>> > > > >
>> > > > > > Thanks Aljoscha and all others who made contributions to FLINK
>> 1.8.0.
>> > > > > > Looking forward to FLINK 1.9.0.
>> > > > > >
>> > > > > > Regards,
>> > > > > > Shaoxuan
>> > > > > >
>> > > > > > On Wed, Apr 10, 2019 at 4:31 PM Aljoscha Krettek <
>> aljos...@apache.org >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > The Apache Flink community is very happy to announce the
>> release of
>> > > > > > Apache
>> > > > > > > Flink 1.8.0, which is the next major release.
>> > > > > > >
>> > > > > > > Apache Flink® is an open-source stream processing framework
>> for
>> > > > > > > distributed, high-performing, always-available, and accurate
>> data
>> > > > > > streaming
>> > > > > > > applications.
>> > > > > > >
>> > > > > > > The release is available for download at:
>> > > > > > > https://flink.apache.org/downloads.html <
>> https://flink.apache.org/downloads.html>
>> > > > > > >
>> > > > > > > Please check out the release blog post for an overview of the
>> > > > > > improvements
>> > > > > > > for this bugfix release:
>> > > > > > > https://flink.apache.org/news/2019/04/09/release-1.8.0.html <
>> https://flink.apache.org/news/2019/04/09/release-1.8.0.html>
>> > > > > > >
>> > > > > > > The full release notes are available in Jira:
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12344274
>> <
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12344274
>> >
>> > > > > > >
>> > > > > > > We would like to thank all contributors of the Apache Flink
>> community
>> > > > > who
>> > > > > > > made this release possible!
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > > Aljoscha
>> > > > > >
>> > > > >
>> > > >
>>
>>

Re: [VOTE] Release 1.8.0, release candidate #4

[Richard Deurwaarder]

Hello @Aljoscha and @Rong,

I've described the problem in the mailing list[1] and on stackoverflow[2]
before. But the gist is: If there's a firewall between the yarn cluster and
the machine submitting the job, we need to be able to set a fixed port (or
range of ports) for REST communication with the jobmanager.

It is a regression in the sense that on 1.5 (and 1.6 I believe?) it was
possible to work around this by using the legacy mode (non flip-6), but on
1.7 and now 1.8 this is not possible.

I've created FLINK-12075 <https://issues.apache.org/jira/browse/FLINK-12075>
for it, I have not made it blocking yet as it is not strictly a regression
with regards to 1.7. Perhaps you guys can better determine if you want this
added in RC5.

Regards,

Richard

[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Submitting-job-to-Flink-on-yarn-timesout-on-flip-6-1-5-x-td26199.html#a26383
[2] https://stackoverflow.com/q/54771637/988324

On Sat, Mar 30, 2019 at 7:24 PM Rong Rong  wrote:

> Hi @Aljoscha,
>
> Based on the previous commit [1] that adds the random port selection code,
> it seems like the important part is to unset whatever 'rest.port' setting
> previously done. I don't think the current way of setting the BIND_PORT
> actually overrides any existing PORT setting. However, I wasn't able to
> find any test that is related, maybe @Till can provide more insight here?
>
> Maybe @Richard can provide more detail on the YARN run command used to
> reproduce the problem?
>
> Thanks,
> Rong
>
> [1]
>
> https://github.com/apache/flink/commit/dbe0e8286d76a5facdb49589b638b87dbde80178#diff-487838863ab693af7008f04cb3359be3R117
>
> On Sat, Mar 30, 2019 at 5:51 AM Aljoscha Krettek 
> wrote:
>
> > @Richard Did this work for you previously? From the change, it seems that
> > the port was always set to 0 on YARN even before.
> >
> > > On 28. Mar 2019, at 16:13, Richard Deurwaarder 
> wrote:
> > >
> > > -1 (non-binding)
> > >
> > > - Ran integration tests locally (1000+) of our flink job, all
> succeeded.
> > > - Attempted to run job on hadoop, failed. It failed because we have a
> > > firewall in place and we cannot set the rest port to a specific
> port/port
> > > range.
> > > Unless I am mistaken, it seems like FLINK-11081 broke the possibility
> of
> > > setting a REST port when running on yarn (
> > >
> >
> https://github.com/apache/flink/commit/730eed71ef3f718d61f85d5e94b1060844ca56db#diff-487838863ab693af7008f04cb3359be3R102
> > > )
> > > Code-wise it seems rather straightforward to fix but I am unsure about
> > the
> > > reason why this is hard-coded to 0 and what the impact would be.
> > >
> > > It would benefit us greatly if a fix for this could make it to 1.8.0.
> > >
> > > Regards,
> > >
> > > Richard
> > >
> > > On Thu, Mar 28, 2019 at 9:54 AM Tzu-Li (Gordon) Tai <
> tzuli...@apache.org
> > >
> > > wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> Functional checks:
> > >>
> > >> - Built Flink from source (`mvn clean verify`) locally, with success
> > >> - Ran end-to-end tests locally for 5 times in a loop, no attempts
> failed
> > >> (Hadoop 2.8.4, Scala 2.12)
> > >> - Manually tested state schema evolution for POJO. Besides the tests
> > that
> > >> @Congxian already did, additionally tested evolution cases with POJO
> > >> subclasses + non-registered POJOs.
> > >> - Manually tested migration of Scala stateful jobs that use case
> > classes /
> > >> Scala collections as state types, performing the migration across
> Scala
> > >> 2.11 to Scala 2.12.
> > >> - Reviewed release announcement PR
> > >>
> > >> Misc / legal checks:
> > >>
> > >> - checked checksums and signatures
> > >> - No binaries in source distribution
> > >> - Staging area does not seem to have any missing artifacts
> > >>
> > >> Cheers,
> > >> Gordon
> > >>
> > >> On Thu, Mar 28, 2019 at 4:52 PM Tzu-Li (Gordon) Tai <
> > tzuli...@apache.org>
> > >> wrote:
> > >>
> > >>> @Shaoxuan
> > >>>
> > >>> The drop in the serializerAvro benchmark, as explained earlier in
> > >> previous
> > >>> voting threads of earlier RCs, was due to a slower job initialization
> > >> phase
> > >>> caused by slower deserialization of the AvroSerializer.
> > >>&g

[jira] [Created] (FLINK-12075) Not able to submit jobs on YARN when there's a firewall

[Richard Deurwaarder (JIRA)]

Richard Deurwaarder created FLINK-12075:
---

 Summary: Not able to submit jobs on YARN when there's a firewall
 Key: FLINK-12075
 URL: https://issues.apache.org/jira/browse/FLINK-12075
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Affects Versions: 1.7.2, 1.8.0
Reporter: Richard Deurwaarder


If there is a firewall around the YARN cluster and the machine, submitting 
flink job it is unpractical because new flink clusters start up with random 
ports for REST communication.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release 1.8.0, release candidate #4

[Richard Deurwaarder]

-1 (non-binding)

- Ran integration tests locally (1000+) of our flink job, all succeeded.
- Attempted to run job on hadoop, failed. It failed because we have a
firewall in place and we cannot set the rest port to a specific port/port
range.
Unless I am mistaken, it seems like FLINK-11081 broke the possibility of
setting a REST port when running on yarn (
https://github.com/apache/flink/commit/730eed71ef3f718d61f85d5e94b1060844ca56db#diff-487838863ab693af7008f04cb3359be3R102
)
Code-wise it seems rather straightforward to fix but I am unsure about the
reason why this is hard-coded to 0 and what the impact would be.

It would benefit us greatly if a fix for this could make it to 1.8.0.

Regards,

Richard

On Thu, Mar 28, 2019 at 9:54 AM Tzu-Li (Gordon) Tai 
wrote:

> +1 (binding)
>
> Functional checks:
>
> - Built Flink from source (`mvn clean verify`) locally, with success
> - Ran end-to-end tests locally for 5 times in a loop, no attempts failed
> (Hadoop 2.8.4, Scala 2.12)
> - Manually tested state schema evolution for POJO. Besides the tests that
> @Congxian already did, additionally tested evolution cases with POJO
> subclasses + non-registered POJOs.
> - Manually tested migration of Scala stateful jobs that use case classes /
> Scala collections as state types, performing the migration across Scala
> 2.11 to Scala 2.12.
> - Reviewed release announcement PR
>
> Misc / legal checks:
>
> - checked checksums and signatures
> - No binaries in source distribution
> - Staging area does not seem to have any missing artifacts
>
> Cheers,
> Gordon
>
> On Thu, Mar 28, 2019 at 4:52 PM Tzu-Li (Gordon) Tai 
> wrote:
>
> > @Shaoxuan
> >
> > The drop in the serializerAvro benchmark, as explained earlier in
> previous
> > voting threads of earlier RCs, was due to a slower job initialization
> phase
> > caused by slower deserialization of the AvroSerializer.
> > Piotr also pointed out that after the number of records was increased in
> > the serializer benchmarks, this drop was no longer observable before /
> > after the changes in mid February.
> > IMO, this is not critical as it does not affect the per-record
> performance
> > / throughput, and therefore should not block this release.
> >
> > On Thu, Mar 28, 2019 at 1:08 AM Aljoscha Krettek 
> > wrote:
> >
> >> By now, I'm reasonably sure that the test instabilities on the
> end-to-end
> >> test are only instabilities. I pushed changes to increase timeouts to
> make
> >> the tests more stable. As in any project, there will always be bugs but
> I
> >> think we could release this RC4 and be reasonably sure that it works
> well.
> >>
> >> Now, we only need to have the required number of PMC votes.
> >>
> >> On Wed, Mar 27, 2019, at 07:22, Congxian Qiu wrote:
> >> > +1 （non-binding)
> >> >
> >> > • checked signature and checksum  ok
> >> > • mvn clean package -DskipTests ok
> >> > • Run job on yarn ok
> >> > • Test state migration with POJO type (both heap and rocksdb) ok
> >> > • - 1.6 -> 1.8
> >> > • - 1.7 -> 1.8
> >> > • - 1.8 -> 1.8
> >> >
> >> >
> >> > Best, Congxian
> >> > On Mar 27, 2019, 10:26 +0800, vino yang ,
> wrote:
> >> > > +1 (non-binding)
> >> > >
> >> > > - checked JIRA release note
> >> > > - ran "mvn package -DskipTests"
> >> > > - checked signature and checksum
> >> > > - started a cluster locally and ran some examples in binary
> >> > > - checked web site announcement's PR
> >> > >
> >> > > Best,
> >> > > Vino
> >> > >
> >> > >
> >> > > Xiaowei Jiang  于2019年3月26日周二 下午8:20写道：
> >> > >
> >> > > > +1 (non-binding)
> >> > > >
> >> > > > - checked checksums and GPG files
> >> > > > - build from source successfully- run end-to-end precommit tests
> >> > > > successfully- run end-to-end nightly tests successfully
> >> > > > Xiaowei
> >> > > > On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <
> >> car...@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > +1 (non-binding)
> >> > > >
> >> > > > - Checked release notes: OK
> >> > > > - Checked sums and signatures: OK
> >> > > > - Source release
> >> > > > - contains no binaries: OK
> >> > > > - contains no 1.8-SNAPSHOT references: OK
> >> > > > - build from source: OK (8u101)
> >> > > > - mvn clean verify: OK (8u101)
> >> > > > - Binary release
> >> > > > - no examples appear to be missing
> >> > > > - started a cluster; WebUI reachable, example ran successfully
> >> > > > - end-to-end test (all but K8S and docker ones): OK (8u101)
> >> > > > - Repository appears to contain all expected artifacts
> >> > > >
> >> > > > Best Regards,
> >> > > > Yu
> >> > > >
> >> > > >
> >> > > > On Tue, 26 Mar 2019 at 14:28, Kurt Young 
> wrote:
> >> > > >
> >> > > > > +1 (non-binding)
> >> > > > >
> >> > > > > Checked items:
> >> > > > > - checked checksums and GPG files
> >> > > > > - verified that the source archives do not contains any binaries
> >> > > > > - checked that all POM files point to the same version
> >> > > > > - build from source successfully
> >> > > > >
> >> > > > > Best,
> >> > > > > Kurt
> >> > > > >
> 

Re: Please review : Re: PubSub connector (FLINK-9311)

[Richard Deurwaarder]

Hello everyone,

To improve performance we have changed some parts of the code:
* Previously the same threads used to poll PubSub we're used to actually
run the flink pipeline, this caused some issues in the PubSub SDK. We've
changed this so that the PubSub threads are only used to poll PubSub and
the flink pipeline runs in the same thread as the run() method.
* We now expose PubSub options so we are able to tune the PubSub source
functions when the defaults don't give the best performance.

I did run into one issue that we are currently unable to fix outside of
flink when rescaling the job to a lower parallelism:
We use the MessageAcknowledgingSourceBase and this had been designed for
RabbitMQ to run with a parallelism of 1. For PubSub we would like a higher
parallelism and fallback to ATLEAST_ONCE.
When rescaling back to a lower parallelism we need to combine state of 2
tasks but the MessageAcknowledingSourceBase explicitly does not support
this in the initializeState() function.
I've added this commit:
https://github.com/Xeli/flink/commit/698b6f1c802427f940f2f550796539fbfa4b5dfa
to
fix this, this should not change anything for RabbitMQ but does allow
rescaling when needed.


That last issue is not possible for us to fix outside of flink, therefore
we would really like to see this added to 1.7 :)
If it remains difficult to find Google Cloud Platform users to review this,
would it be an option to add the PubSub connector as a 'beta' feature?
Perhaps this would make it more visible to the community?

Best,

Richard

On Fri, Sep 14, 2018 at 4:01 PM Aljoscha Krettek 
wrote:

> Hi Niels and Richard,
>
> I would be very happy about having a PubSub connector in Flink. Having it
> in Flink means that you don't have manual effort for tracking API changes
> and I think having a production user is incentive enough for them (you) to
> maintain the connector.
>
> I'm afraid we don't have much PubSub knowledge in the Flink community but
> I will try and talk to some folks to have this reviewed.
>
> Best,
> Aljoscha
>
> > On 14. Sep 2018, at 15:40, Niels Basjes  wrote:
> >
> > Hi all,
> >
> > We (Richard and I) would really appreciate it if you guys could review
> > the new feature we created (see below).
> > It is something we really need in production and thought it would be
> > best if it can be a native part of the Flink toolset.
> > Please indicate what we need to change/improve in order to get this
> committed.
> >
> > Thank you.
> >
> > Niels Basjes
> >
> > On Mon, Sep 10, 2018 at 10:12 AM Richard Deurwaarder 
> wrote:
> >>
> >> Hello everyone,
> >>
> >> A while back I opened this Jira issue:
> >> https://issues.apache.org/jira/browse/FLINK-9311. It is regarding a new
> >> Flink connector for Google PubSub.
> >>
> >> At Bol.com, the company I work for, we are going to use this connector
> to
> >> do our financial processing. My colleague, Niels Basjes, and I have
> >> finished implementing this and we are about ready to start running it in
> >> production.
> >>
> >> We would like to donate this code and have opened a pull request (
> >> https://github.com/apache/flink/pull/6594). The pull request contains:
> >>
> >>   - a SourceFunction (with 2 test versions)
> >>   - a SinkFunction
> >>   - an example application
> >>   - and End-to-End tests using a docker container
> >>
> >> Yanghua has taken a glance at the code and already provided us with some
> >> feedback. We would like to invite others to provide additional feedback.
> >>
> >> I hope you find this useful and will consider merging the PR!
> >>
> >> Best,
> >>
> >> Richard Deurwaarder
> >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
>
>

Re: Please review : Re: PubSub connector (FLINK-9311)

[Richard Deurwaarder]

Hello everyone,

To improve performance we have changed some parts of the code:
* Previously the same threads used to poll PubSub we're used to actually
run the flink pipeline, this caused some issues in the PubSub SDK. We've
changed this so that the PubSub threads are only used to poll PubSub and
the flink pipeline runs in the same thread as the run() method.
* We now expose PubSub options so we are able to tune the PubSub source
functions when the defaults don't give the best performance.

I did run into one issue that we are currently unable to fix outside of
flink when rescaling the job to a lower parallelism:
We use the MessageAcknowledgingSourceBase and this had been designed for
RabbitMQ to run with a parallelism of 1. For PubSub we would like a higher
parallelism and fallback to ATLEAST_ONCE.
When rescaling back to a lower parallelism we need to combine state of 2
tasks but the MessageAcknowledingSourceBase explicitly does not support
this in the initializeState() function.
I've added this commit:
https://github.com/Xeli/flink/commit/698b6f1c802427f940f2f550796539fbfa4b5dfa
to fix this, this should not change anything for RabbitMQ but does allow
rescaling when needed.


That last issue is not possible for us to fix outside of flink, therefore
we would really like to see this added to 1.7 :)
If it remains difficult to find Google Cloud Platform users to review this
would it be an option to add the PubSub connector as a 'beta' feature?
Perhaps this would make it more visible to the community?

Best,

Richard

On Fri, Sep 14, 2018 at 4:01 PM Aljoscha Krettek 
wrote:

> Hi Niels and Richard,
>
> I would be very happy about having a PubSub connector in Flink. Having it
> in Flink means that you don't have manual effort for tracking API changes
> and I think having a production user is incentive enough for them (you) to
> maintain the connector.
>
> I'm afraid we don't have much PubSub knowledge in the Flink community but
> I will try and talk to some folks to have this reviewed.
>
> Best,
> Aljoscha
>
> > On 14. Sep 2018, at 15:40, Niels Basjes  wrote:
> >
> > Hi all,
> >
> > We (Richard and I) would really appreciate it if you guys could review
> > the new feature we created (see below).
> > It is something we really need in production and thought it would be
> > best if it can be a native part of the Flink toolset.
> > Please indicate what we need to change/improve in order to get this
> committed.
> >
> > Thank you.
> >
> > Niels Basjes
> >
> > On Mon, Sep 10, 2018 at 10:12 AM Richard Deurwaarder 
> wrote:
> >>
> >> Hello everyone,
> >>
> >> A while back I opened this Jira issue:
> >> https://issues.apache.org/jira/browse/FLINK-9311. It is regarding a new
> >> Flink connector for Google PubSub.
> >>
> >> At Bol.com, the company I work for, we are going to use this connector
> to
> >> do our financial processing. My colleague, Niels Basjes, and I have
> >> finished implementing this and we are about ready to start running it in
> >> production.
> >>
> >> We would like to donate this code and have opened a pull request (
> >> https://github.com/apache/flink/pull/6594). The pull request contains:
> >>
> >>   - a SourceFunction (with 2 test versions)
> >>   - a SinkFunction
> >>   - an example application
> >>   - and End-to-End tests using a docker container
> >>
> >> Yanghua has taken a glance at the code and already provided us with some
> >> feedback. We would like to invite others to provide additional feedback.
> >>
> >> I hope you find this useful and will consider merging the PR!
> >>
> >> Best,
> >>
> >> Richard Deurwaarder
> >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
>
>

PubSub connector (FLINK-9311)

[Richard Deurwaarder]

Hello everyone,

A while back I opened this Jira issue:
https://issues.apache.org/jira/browse/FLINK-9311. It is regarding a new
Flink connector for Google PubSub.

At Bol.com, the company I work for, we are going to use this connector to
do our financial processing. My colleague, Niels Basjes, and I have
finished implementing this and we are about ready to start running it in
production.

We would like to donate this code and have opened a pull request (
https://github.com/apache/flink/pull/6594). The pull request contains:

   - a SourceFunction (with 2 test versions)
   - a SinkFunction
   - an example application
   - and End-to-End tests using a docker container

Yanghua has taken a glance at the code and already provided us with some
feedback. We would like to invite others to provide additional feedback.

I hope you find this useful and will consider merging the PR!

Best,

Richard Deurwaarder

[jira] [Created] (FLINK-9311) PubSub connector

[Richard Deurwaarder (JIRA)]

Richard Deurwaarder created FLINK-9311:
--

 Summary: PubSub connector
 Key: FLINK-9311
 URL: https://issues.apache.org/jira/browse/FLINK-9311
 Project: Flink
  Issue Type: New Feature
  Components: Streaming Connectors
Reporter: Richard Deurwaarder


I would like start adding some google cloud connectors starting with a PubSub 
Source. I have a basic implementation ready but I want it to be able to:
 * easily scale up (should I have it extend RichParallelSourceFunction for 
this?)
 * Make it easier to provide the google cloud credentials. This would require 
being able to send some json string / ServiceAccount to the nodes when starting 
up this source.

Could this be something that would be useful for others and added to the flink 
connectors repo?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)