[jira] [Created] (FLINK-9440) Allow cancelation and reset of timers

2018-05-25 Thread Elias Levy (JIRA) 
Elias Levy created FLINK-9440:
-

 Summary: Allow cancelation and reset of timers
 Key: FLINK-9440
 URL: https://issues.apache.org/jira/browse/FLINK-9440
 Project: Flink
  Issue Type: Bug
  Components: Streaming
Affects Versions: 1.4.2
Reporter: Elias Levy


Currently the {{TimerService}} allows one to register timers, but it is not 
possible to delete a timer or to reset a timer to a new value.  If one wishes 
to reset a timer, one must also handle the previous inserted timer callbacks 
and ignore them.

I would be useful if the API allowed one to remove and reset timers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] Apache Flink 1.5.0 release

2018-05-25 Thread Till Rohrmann 
Quick update: I had to update the date of the release blog post which also
changed the URL. It can now be found here:

http://flink.apache.org/news/2018/05/25/release-1.5.0.html

Cheers,
Till

On Fri, May 25, 2018 at 7:03 PM, Hao Sun  wrote:

> This is great. Thanks for the effort to get this out!
>
> On Fri, May 25, 2018 at 9:47 AM Till Rohrmann 
> wrote:
>
>> The Apache Flink community is very happy to announce the release of
>> Apache Flink 1.5.0.
>>
>> Apache Flink® is an open-source stream processing framework for
>> distributed, high-performing, always-available, and accurate data streaming
>> applications.
>>
>> The release is available for download at:
>>
>> https://flink.apache.org/downloads.html
>>
>> Please check out the release blog post for an overview of the new
>> features and improvements and the list of contributors:
>>
>> http://flink.apache.org/news/2018/05/18/release-1.5.0.html
>>
>> The full release notes are available in Jira:
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12315522=12341764
>>
>> I would like to thank all contributors for working very hard on making
>> this release a success!
>>
>> Best,
>> Till
>>
>

Re: [VOTE] Enable GitBox integration (#2)

2018-05-25 Thread Rong Rong 
+1

On Fri, May 25, 2018 at 4:14 AM, Qi Yu  wrote:

> +1
>
> > 在 2018年5月25日，下午7:12，Till Rohrmann  写道：
> >
> > +1
> >
> > On Wed, May 23, 2018 at 2:33 PM, Stefan Richter <
> s.rich...@data-artisans.com
> >> wrote:
> >
> >> +1
> >>
> >>> Am 23.05.2018 um 14:31 schrieb Stephan Ewen :
> >>>
> >>> +1
> >>>
> >>> On Wed, May 23, 2018 at 11:11 AM, Fabian Hueske 
> >> wrote:
> >>>
>  +1
> 
>  2018-05-23 8:49 GMT+02:00 Aljoscha Krettek :
> 
> > +1
> >
> >> On 22. May 2018, at 15:36, Thomas Weise  wrote:
> >>
> >> +1
> >>
> >>
> >> On Tue, May 22, 2018 at 2:37 AM, Timo Walther 
> > wrote:
> >>
> >>> +1
> >>>
> >>> Am 22.05.18 um 10:49 schrieb Ted Yu:
> >>>
> >>> +1
>   Original message From: Chesnay Schepler <
>  ches...@apache.org> Date: 5/22/18  1:12 AM  (GMT-08:00) To:
>  dev@flink.apache.org Subject: [VOTE] Enable GitBox integration
> (#2)
>  Hello,
> 
>  since no concerns have been raised in the discussion about
> enabling
>  GitBox [1] I'm opening this vote to make things official.
> 
>  Please vote on enabling GitBox integration for the flink and
>  flink-web
>  repositories as follows:
> 
>  [ ] +1, Approve GitBox
>  [ ] -1, Do not approve GitBox (please provide specific comments)
> 
>  The vote will be open for at least 72 hours. It is adopted by
>  majority
>  approval, with at least 3 PMC affirmative votes.
> 
>  This is the second attempt for this vote. The previous vote was
>  cancelled due to the flink-web repository settings. Changes to the
>  previous vote are highlighted in bold.
> 
>  If accepted I will file a ticket with INFRA to enable GitBox with
> >> the
>  following settings:
> 
>  flink:
> 
>  * no wiki
>  * no issues
>  * no projects
>  * no merge commit button (rebase/squash merge button still
>  enabled)
>  * protected branches [2]: master, release-1.[0-5] (the latter we
> > will
>    have to update with each release)
> 
>  flink-web
> 
>  * no wiki
>  **no issues*
>  * no projects
>  * no merge commit button (rebase/squash merge button still
>  enabled)
>  * protected branch: asf-site
>  * default branch: asf-site (while we're at it...)
> 
>  [1]
>  http://apache-flink-mailing-list-archive.1008284.n3.nabble.
>  com/DISCUSS-GitBox-td22328.html
>  [2] https://help.github.com/articles/about-protected-branches/
> 
> 
> 
> >>>
> >
> >
> 
> >>
> >>
>
>

[ANNOUNCE] Apache Flink 1.5.0 release

2018-05-25 Thread Till Rohrmann 
The Apache Flink community is very happy to announce the release of Apache
Flink 1.5.0.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:

https://flink.apache.org/downloads.html

Please check out the release blog post for an overview of the new features
and improvements and the list of contributors:

http://flink.apache.org/news/2018/05/18/release-1.5.0.html

The full release notes are available in Jira:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12341764

I would like to thank all contributors for working very hard on making this
release a success!

Best,
Till

[RESULT] [VOTE] Release 1.5.0, release candidate #6

2018-05-25 Thread Till Rohrmann 
I'm happy to announce that we have unanimously approved this release.

There are 7 approving votes, 4 of which are binding:
- Piotrek (non-binding)
- Aljoscha (binding)
- Timo (binding)
- Tzu-Li (binding)
- Till (binding)
- Gary (non-binding)
- Thomas (non-binding)

There are no disapproving votes.

Thanks everyone for the hard work and help making this release possible!

Re: [VOTE] Release 1.5.0, release candidate #6

2018-05-25 Thread Thomas Weise 
+1 (non-binding)

- checked signatures and hashes
- run Kinesis consumer test

Regarding the rate of issues being found: I would credit some of it to
increased coverage and due diligence and some to to the nature of changes
in the release.

It is probably expected that a 1.5.1 release will follow soon and the
effort for the friendly RM hopefully not that different from rolling 1.5.0
RCs :)

Cheers,
Thomas


On Thu, May 24, 2018 at 9:51 AM, Till Rohrmann  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #6 for the version 1.5.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint 1F302569A96CFFD5 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-1.5.0-rc6" [5],
> * PR to update the community web site [6]
>
> Please use this document for coordinating testing efforts: [7]
>
> The voting periods ends tomorrow at 5pm CET. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Your friendly Release Manager
>
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12315522=12341764
> [2] http://people.apache.org/~trohrmann/flink-1.5.0-rc6/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/
> orgapacheflink-1161/
> [5]
> https://git-wip-us.apache.org/repos/asf?p=flink.git;a=commit;h=
> c61b108b8eaa22eac3dc492b3c95c22b4177003f
> [6] https://github.com/apache/flink-web/pull/106
> [7]
> https://docs.google.com/document/d/10KDBLzt25Z44pdZBSm8MTeKldc4UT
> UAbAynfuVQWOt0/edit?usp=sharing
>
> Pro-tip: you can create a settings.xml file with these contents:
>
> 
> 
>   flink-1.5.0
> 
> 
>   
> flink-1.5.0
> 
>   
> flink-1.5.0
> 
>
> https://repository.apache.org/content/repositories/orgapacheflink-1161/
> 
>   
>   
> archetype
> 
>
> https://repository.apache.org/content/repositories/orgapacheflink-1161/
> 
>   
> 
>   
> 
> 
>
> And reference that in you maven commands via --settings
> path/to/settings.xml. This is useful for creating a quickstart based on the
> staged release and for building against the staged jars.
>

Re: [VOTE] Release 1.5.0, release candidate #6

2018-05-25 Thread Till Rohrmann 
The voting time has passed and I'm happy to announce that we've collected
enough votes to release this RC as Flink 1.5.0.

+1 votes:
- Piotrek (non-binding)
- Aljoscha (binding)
- Timo (binding)
- Tzu-Li (binding)
- Till (binding)
- Gary (non-binding)

That's 6 votes, 4 binding. No 0 or -1 votes.

Thanks a lot, everyone, for testing and making sure that this will be a good
 release! I'll send out a separate announcement mail and push out the
release artifacts and update the website now.

On Fri, May 25, 2018 at 5:06 PM, Gary Yao  wrote:

> +1 (non-binding)
>
> I have tested HA capabilities on YARN (Hadoop 2.8.3).
>
> Best,
> Gary
>
> On Fri, May 25, 2018 at 4:54 PM, Tzu-Li (Gordon) Tai 
> wrote:
>
> > +1
> >
> > - Verified signatures and hashes
> > - Built from source with Hadoop 2.8.1 and Scala 2.11.7, tests pass
> locally
> > - No binaries in source archives; no missing artifacts
> > - To cover the extra Kafka connector bug fixes that were added in between
> > RCs (FLINK-9349 and FLINK-9295), I executed Kafka end-to-end test
> non-stop
> > for over 3 hours. All executions passed.
> > - Did one pass over the run-nightly-tests.sh and pre-commit-tests.sh, no
> > failures
> >
> > Cheers,
> > Gordon
> > On 25 May 2018 at 10:41:00 PM, Timo Walther (twal...@apache.org) wrote:
> >
> > +1
> >
> > - I build the release locally with the minor issues mentioned in the
> > last thread
> > - I executed a couple of table programs on a local cluster
> > - Ran some end-to-end tests locally
> >
> > @Piotr: Given the amount of changes that went into this release it is
> > natural to find a lot of bugs. We have also increased our testing
> > efforts (e.g. by having more automated tests) this time.
> >
> > Timo
> >
> > Am 25.05.18 um 14:12 schrieb Aljoscha Krettek:
> > > +1
> > >
> > > - verified signature and hashes
> > > - ran an example on a kerberized Hadoop 2.8.3 cluster, both with kinit
> > and by using keytab
> > >
> > >> On 25. May 2018, at 11:56, Piotr Nowojski 
> > wrote:
> > >>
> > >> Mild +1 from me. I’m concerned about the rate of bugs that we are
> > discovering for each RC, which suggests that there are still some release
> > blockers out there (but I’m not aware of any right now).
> > >>
> > >> Piotrek
> > >>
> > >>> On 25 May 2018, at 11:25, Till Rohrmann 
> > wrote:
> > >>>
> > >>> +1
> > >>>
> > >>> - Checked checksums and GPG files
> > >>> - Verified that source archives do not contain any binaries
> > >>> - Built Flink with Hadoop 2.8.1 and Scala 2.11.7 from source release
> > >>> - Verified LICENSE and NOTICE file: The LICENSE file contains
> > unnecessary
> > >>> entries for jline-reader and jline-terminal
> > >>> - Checked licenses of newly added dependencies
> > >>> - Checked pom files
> > >>> - Read README.md
> > >>> - Run manual tests in flink-tests: CheckForbiddenMethodsUsage fails
> > >>> because org.apache.flink.queryablestate.messages.
> KvStateRequest.serialize()
> >
> > >>> uses getBytes without charset. This is non-blocking because it was
> > >>> introduced with 1.4 or before
> > >>> - Checked builds with SBT and the SBT quickstarts
> > >>> - Executed the run-nightly-tests.sh for 10 hours without failures
> > >>> - Executed Jepsen tests without failures
> > >>>
> > >>> There is a known problem when enabling SSL encryption which can lead
> > to
> > >>> failures [1]. Since there is a workaround for this problem, I would
> > propose
> > >>> to not block the release on it and fix it with Flink 1.5.1.
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/FLINK-9437
> > >>>
> > >>> Cheers,
> > >>> Till
> > >>>
> > >>>
> > >>> On Thu, May 24, 2018 at 6:51 PM, Till Rohrmann  >
> > wrote:
> > >>>
> >  Hi everyone,
> > 
> >  Please review and vote on the release candidate #6 for the version
> > 1.5.0,
> >  as follows:
> >  [ ] +1, Approve the release
> >  [ ] -1, Do not approve the release (please provide specific
> > comments)
> > 
> > 
> >  The complete staging area is available for your review, which
> > includes:
> >  * JIRA release notes [1],
> >  * the official Apache source release and binary convenience releases
> > to be
> >  deployed to dist.apache.org [2], which are signed with the key with
> >  fingerprint 1F302569A96CFFD5 [3],
> >  * all artifacts to be deployed to the Maven Central Repository [4],
> >  * source code tag "release-1.5.0-rc6" [5],
> >  * PR to update the community web site [6]
> > 
> >  Please use this document for coordinating testing efforts: [7]
> > 
> >  The voting periods ends tomorrow at 5pm CET. It is adopted by
> > majority
> >  approval, with at least 3 PMC affirmative votes.
> > 
> >  Thanks,
> >  Your friendly Release Manager
> > 
> >  [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> >  projectId=12315522=12341764
> >  [2]

Re: [VOTE] Release 1.5.0, release candidate #6

2018-05-25 Thread Gary Yao 
+1 (non-binding)

I have tested HA capabilities on YARN (Hadoop 2.8.3).

Best,
Gary

On Fri, May 25, 2018 at 4:54 PM, Tzu-Li (Gordon) Tai 
wrote:

> +1
>
> - Verified signatures and hashes
> - Built from source with Hadoop 2.8.1 and Scala 2.11.7, tests pass locally
> - No binaries in source archives; no missing artifacts
> - To cover the extra Kafka connector bug fixes that were added in between
> RCs (FLINK-9349 and FLINK-9295), I executed Kafka end-to-end test non-stop
> for over 3 hours. All executions passed.
> - Did one pass over the run-nightly-tests.sh and pre-commit-tests.sh, no
> failures
>
> Cheers,
> Gordon
> On 25 May 2018 at 10:41:00 PM, Timo Walther (twal...@apache.org) wrote:
>
> +1
>
> - I build the release locally with the minor issues mentioned in the
> last thread
> - I executed a couple of table programs on a local cluster
> - Ran some end-to-end tests locally
>
> @Piotr: Given the amount of changes that went into this release it is
> natural to find a lot of bugs. We have also increased our testing
> efforts (e.g. by having more automated tests) this time.
>
> Timo
>
> Am 25.05.18 um 14:12 schrieb Aljoscha Krettek:
> > +1
> >
> > - verified signature and hashes
> > - ran an example on a kerberized Hadoop 2.8.3 cluster, both with kinit
> and by using keytab
> >
> >> On 25. May 2018, at 11:56, Piotr Nowojski 
> wrote:
> >>
> >> Mild +1 from me. I’m concerned about the rate of bugs that we are
> discovering for each RC, which suggests that there are still some release
> blockers out there (but I’m not aware of any right now).
> >>
> >> Piotrek
> >>
> >>> On 25 May 2018, at 11:25, Till Rohrmann 
> wrote:
> >>>
> >>> +1
> >>>
> >>> - Checked checksums and GPG files
> >>> - Verified that source archives do not contain any binaries
> >>> - Built Flink with Hadoop 2.8.1 and Scala 2.11.7 from source release
> >>> - Verified LICENSE and NOTICE file: The LICENSE file contains
> unnecessary
> >>> entries for jline-reader and jline-terminal
> >>> - Checked licenses of newly added dependencies
> >>> - Checked pom files
> >>> - Read README.md
> >>> - Run manual tests in flink-tests: CheckForbiddenMethodsUsage fails
> >>> because 
> >>> org.apache.flink.queryablestate.messages.KvStateRequest.serialize()
>
> >>> uses getBytes without charset. This is non-blocking because it was
> >>> introduced with 1.4 or before
> >>> - Checked builds with SBT and the SBT quickstarts
> >>> - Executed the run-nightly-tests.sh for 10 hours without failures
> >>> - Executed Jepsen tests without failures
> >>>
> >>> There is a known problem when enabling SSL encryption which can lead
> to
> >>> failures [1]. Since there is a workaround for this problem, I would
> propose
> >>> to not block the release on it and fix it with Flink 1.5.1.
> >>>
> >>> [1] https://issues.apache.org/jira/browse/FLINK-9437
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>>
> >>> On Thu, May 24, 2018 at 6:51 PM, Till Rohrmann 
> wrote:
> >>>
>  Hi everyone,
> 
>  Please review and vote on the release candidate #6 for the version
> 1.5.0,
>  as follows:
>  [ ] +1, Approve the release
>  [ ] -1, Do not approve the release (please provide specific
> comments)
> 
> 
>  The complete staging area is available for your review, which
> includes:
>  * JIRA release notes [1],
>  * the official Apache source release and binary convenience releases
> to be
>  deployed to dist.apache.org [2], which are signed with the key with
>  fingerprint 1F302569A96CFFD5 [3],
>  * all artifacts to be deployed to the Maven Central Repository [4],
>  * source code tag "release-1.5.0-rc6" [5],
>  * PR to update the community web site [6]
> 
>  Please use this document for coordinating testing efforts: [7]
> 
>  The voting periods ends tomorrow at 5pm CET. It is adopted by
> majority
>  approval, with at least 3 PMC affirmative votes.
> 
>  Thanks,
>  Your friendly Release Manager
> 
>  [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>  projectId=12315522=12341764
>  [2] http://people.apache.org/~trohrmann/flink-1.5.0-rc6/
>  [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>  [4] https://repository.apache.org/content/repositories/
>  orgapacheflink-1161/
>  [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=commit;h=
>  c61b108b8eaa22eac3dc492b3c95c22b4177003f
>  [6] https://github.com/apache/flink-web/pull/106
>  [7] https://docs.google.com/document/d/10KDBLzt25Z44pdZBSm8MTeKldc4UT
>
>  UAbAynfuVQWOt0/edit?usp=sharing
> 
>  Pro-tip: you can create a settings.xml file with these contents:
> 
>  
>  
>  flink-1.5.0
>  
>  
>  
>  flink-1.5.0
>  
>  
>  flink-1.5.0
>  
>  https://repository.apache.org/content/repositories/
>  orgapacheflink-1161/
>  
>  
>

Re: [VOTE] Release 1.5.0, release candidate #6

2018-05-25 Thread Tzu-Li (Gordon) Tai 
+1

- Verified signatures and hashes
- Built from source with Hadoop 2.8.1 and Scala 2.11.7, tests pass locally
- No binaries in source archives; no missing artifacts
- To cover the extra Kafka connector bug fixes that were added in between RCs 
(FLINK-9349 and FLINK-9295), I executed Kafka end-to-end test non-stop for over 
3 hours. All executions passed.
- Did one pass over the run-nightly-tests.sh and pre-commit-tests.sh, no 
failures

Cheers,
Gordon
On 25 May 2018 at 10:41:00 PM, Timo Walther (twal...@apache.org) wrote:

+1  

- I build the release locally with the minor issues mentioned in the  
last thread  
- I executed a couple of table programs on a local cluster  
- Ran some end-to-end tests locally  

@Piotr: Given the amount of changes that went into this release it is  
natural to find a lot of bugs. We have also increased our testing  
efforts (e.g. by having more automated tests) this time.  

Timo  

Am 25.05.18 um 14:12 schrieb Aljoscha Krettek:  
> +1  
>  
> - verified signature and hashes  
> - ran an example on a kerberized Hadoop 2.8.3 cluster, both with kinit and by 
> using keytab  
>  
>> On 25. May 2018, at 11:56, Piotr Nowojski  wrote:  
>>  
>> Mild +1 from me. I’m concerned about the rate of bugs that we are 
>> discovering for each RC, which suggests that there are still some release 
>> blockers out there (but I’m not aware of any right now).  
>>  
>> Piotrek  
>>  
>>> On 25 May 2018, at 11:25, Till Rohrmann  wrote:  
>>>  
>>> +1  
>>>  
>>> - Checked checksums and GPG files  
>>> - Verified that source archives do not contain any binaries  
>>> - Built Flink with Hadoop 2.8.1 and Scala 2.11.7 from source release  
>>> - Verified LICENSE and NOTICE file: The LICENSE file contains unnecessary  
>>> entries for jline-reader and jline-terminal  
>>> - Checked licenses of newly added dependencies  
>>> - Checked pom files  
>>> - Read README.md  
>>> - Run manual tests in flink-tests: CheckForbiddenMethodsUsage fails  
>>> because org.apache.flink.queryablestate.messages.KvStateRequest.serialize() 
>>>  
>>> uses getBytes without charset. This is non-blocking because it was  
>>> introduced with 1.4 or before  
>>> - Checked builds with SBT and the SBT quickstarts  
>>> - Executed the run-nightly-tests.sh for 10 hours without failures  
>>> - Executed Jepsen tests without failures  
>>>  
>>> There is a known problem when enabling SSL encryption which can lead to  
>>> failures [1]. Since there is a workaround for this problem, I would propose 
>>>  
>>> to not block the release on it and fix it with Flink 1.5.1.  
>>>  
>>> [1] https://issues.apache.org/jira/browse/FLINK-9437  
>>>  
>>> Cheers,  
>>> Till  
>>>  
>>>  
>>> On Thu, May 24, 2018 at 6:51 PM, Till Rohrmann  
>>> wrote:  
>>>  
 Hi everyone,  
  
 Please review and vote on the release candidate #6 for the version 1.5.0,  
 as follows:  
 [ ] +1, Approve the release  
 [ ] -1, Do not approve the release (please provide specific comments)  
  
  
 The complete staging area is available for your review, which includes:  
 * JIRA release notes [1],  
 * the official Apache source release and binary convenience releases to be 
  
 deployed to dist.apache.org [2], which are signed with the key with  
 fingerprint 1F302569A96CFFD5 [3],  
 * all artifacts to be deployed to the Maven Central Repository [4],  
 * source code tag "release-1.5.0-rc6" [5],  
 * PR to update the community web site [6]  
  
 Please use this document for coordinating testing efforts: [7]  
  
 The voting periods ends tomorrow at 5pm CET. It is adopted by majority  
 approval, with at least 3 PMC affirmative votes.  
  
 Thanks,  
 Your friendly Release Manager  
  
 [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?  
 projectId=12315522=12341764  
 [2] http://people.apache.org/~trohrmann/flink-1.5.0-rc6/  
 [3] https://dist.apache.org/repos/dist/release/flink/KEYS  
 [4] https://repository.apache.org/content/repositories/  
 orgapacheflink-1161/  
 [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=commit;h=  
 c61b108b8eaa22eac3dc492b3c95c22b4177003f  
 [6] https://github.com/apache/flink-web/pull/106  
 [7] https://docs.google.com/document/d/10KDBLzt25Z44pdZBSm8MTeKldc4UT  
 UAbAynfuVQWOt0/edit?usp=sharing  
  
 Pro-tip: you can create a settings.xml file with these contents:  
  
   
   
 flink-1.5.0  
   
   
   
 flink-1.5.0  
   
   
 flink-1.5.0  
   
 https://repository.apache.org/content/repositories/  
 orgapacheflink-1161/  
   
   
   
 archetype  
   
 https://repository.apache.org/content/repositories/  
 orgapacheflink-1161/  
   
   
   
   
   
   
  
 And reference that in

Re: [VOTE] Release 1.5.0, release candidate #6

2018-05-25 Thread Timo Walther 

+1

- I build the release locally with the minor issues mentioned in the 
last thread

- I executed a couple of table programs on a local cluster
- Ran some end-to-end tests locally

@Piotr: Given the amount of changes that went into this release it is 
natural to find a lot of bugs. We have also increased our testing 
efforts (e.g. by having more automated tests) this time.


Timo

Am 25.05.18 um 14:12 schrieb Aljoscha Krettek:

+1

  - verified signature and hashes
  - ran an example on a kerberized Hadoop 2.8.3 cluster, both with kinit and by 
using keytab


On 25. May 2018, at 11:56, Piotr Nowojski  wrote:

Mild +1 from me. I’m concerned about the rate of bugs that we are discovering 
for each RC, which suggests that there are still some release blockers out 
there (but I’m not aware of any right now).

Piotrek


On 25 May 2018, at 11:25, Till Rohrmann  wrote:

+1

- Checked checksums and GPG files
- Verified that source archives do not contain any binaries
- Built Flink with Hadoop 2.8.1 and Scala 2.11.7 from source release
- Verified LICENSE and NOTICE file: The LICENSE file contains unnecessary
entries for jline-reader and jline-terminal
- Checked licenses of newly added dependencies
- Checked pom files
- Read README.md
- Run manual tests in flink-tests: CheckForbiddenMethodsUsage fails
because org.apache.flink.queryablestate.messages.KvStateRequest.serialize()
uses getBytes without charset. This is non-blocking because it was
introduced with 1.4 or before
- Checked builds with SBT and the SBT quickstarts
- Executed the run-nightly-tests.sh for 10 hours without failures
- Executed Jepsen tests without failures

There is a known problem when enabling SSL encryption which can lead to
failures [1]. Since there is a workaround for this problem, I would propose
to not block the release on it and fix it with Flink 1.5.1.

[1] https://issues.apache.org/jira/browse/FLINK-9437

Cheers,
Till


On Thu, May 24, 2018 at 6:51 PM, Till Rohrmann  wrote:


Hi everyone,

Please review and vote on the release candidate #6 for the version 1.5.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release and binary convenience releases to be
deployed to dist.apache.org [2], which are signed with the key with
fingerprint 1F302569A96CFFD5 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-1.5.0-rc6" [5],
* PR to update the community web site [6]

Please use this document for coordinating testing efforts: [7]

The voting periods ends tomorrow at 5pm CET. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Your friendly Release Manager

[1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
projectId=12315522=12341764
[2] http://people.apache.org/~trohrmann/flink-1.5.0-rc6/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/
orgapacheflink-1161/
[5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=commit;h=
c61b108b8eaa22eac3dc492b3c95c22b4177003f
[6] https://github.com/apache/flink-web/pull/106
[7] https://docs.google.com/document/d/10KDBLzt25Z44pdZBSm8MTeKldc4UT
UAbAynfuVQWOt0/edit?usp=sharing

Pro-tip: you can create a settings.xml file with these contents:



flink-1.5.0



   flink-1.5.0
   
 
   flink-1.5.0
   
   https://repository.apache.org/content/repositories/
orgapacheflink-1161/
   
 
 
   archetype
   
   https://repository.apache.org/content/repositories/
orgapacheflink-1161/
   
 
   




And reference that in you maven commands via --settings
path/to/settings.xml. This is useful for creating a quickstart based on the
staged release and for building against the staged jars.

[PROPOSAL] Improving Flink’s timer management for large state

2018-05-25 Thread Stefan Richter 
Hi,

I am currently planning how to improve Flink’s timer management for large 
state. In particular, I would like to introduce timer state that is managed in 
RocksDB and also to improve the capabilities of the heap-based timer service, 
e.g. support for asynchronous checkpoints. You can find a short outline of my 
planned approach in this document: 

https://docs.google.com/document/d/1XbhJRbig5c5Ftd77d0mKND1bePyTC26Pz04EvxdA7Jc/edit?usp=sharing

As always, your questions, feedback, and comments are highly appreciated.

Best,
Stefan

Re: Increasing Disk Read Throughput and IOPS

2018-05-25 Thread Andrey Zagrebin 
Hi,

I just wanted to add that if you are using EBS you could consider to switch to 
IO provisioned type of it (io1: 
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html 
) if 
it is ok from the cost prospective. There is no burst credit but a steady IOPS 
rate can be provisioned which is higher than the baseline of general purpose 
type gp2 (for 100GB: 5k IOPS of io1 vs 0.3k IOPS of gp2 baseline). It might 
speed up background compaction and improve read performance.

In general, EBS fault tolerance does not have a lot of benefit for the current 
version of Flink. I agree to consider instance ephemeral ssd storage instead 
which seems to be anyways couple of times more performant on bigger rocksdb.

Andrey

> On 25 May 2018, at 10:52, Stefan Richter  wrote:
> 
> One more thing, I am aware of one older thread that might be interesting for 
> you about RocksDB backend and EBS: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/checkpoint-stuck-with-rocksdb-statebackend-and-s3-filesystem-td18679.html
>  
> 
> 
>> Am 25.05.2018 um 09:59 schrieb Stefan Richter > >:
>> 
>> Hi,
>> 
>> if the problem is seemingly from reads, I think incremental checkpoints are 
>> less likely to cause the problem. What Flink version are you using? Since 
>> you mentioned the use of map state, what comes to my mind as a potential 
>> cause is described in this issue 
>> https://issues.apache.org/jira/browse/FLINK-8639 
>>  . This was improved 
>> recently. Does the problem also exist for jobs without map state?
>> 
>> Best,
>> Stefan
>> 
>>> Am 24.05.2018 um 20:25 schrieb Stephan Ewen >> >:
>>> 
>>> One thing that you can always to is disable fsync, because Flink does not 
>>> rely on RocksDBs fsync for persistence.
>>> 
>>> If you disable incremental checkpoints, does that help?
>>> If yes, it could be an issue with too many small SSTable files due to 
>>> incremental checkpoints (an issue we have on the roadmap to fix).
>>> 
>>> On Thu, May 24, 2018 at 3:52 PM, Piotr Nowojski >> > wrote:
>>> Hi,
>>> 
>>> This issue might have something to do with compaction. Problems with 
>>> compaction can especially degrade reads performance (or just increase reads 
>>> IO). Have you tried to further enforce more compactions or change 
>>> CompactionStyle?
>>> 
>>> Have you taken a look on 
>>> org.apache.flink.contrib.streaming.state.PredefinedOptions?
>>> 
>>> Maybe Stefan or Andrey could share more input on this.
>>> 
>>> Piotrek
>>> 
>>> 
>>> > On 22 May 2018, at 08:12, Govindarajan Srinivasaraghavan 
>>> > > wrote:
>>> > 
>>> > Hi All,
>>> > 
>>> > We are running flink in AWS and we are observing a strange behavior. We 
>>> > are using docker containers, EBS for storage and Rocks DB state backend. 
>>> > We have a few map and value states with checkpointing every 30 seconds 
>>> > and incremental checkpointing turned on. The issue we are noticing is the 
>>> > read IOPS and read throughput gradually increases over time and keeps 
>>> > constantly growing. The write throughput and write bytes are not 
>>> > increasing as much as reads. The checkpoints are written to a durable NFS 
>>> > storage. We are not sure what is causing this constant increase in read 
>>> > throughput but due to which we are running out of EBS burst balance and 
>>> > need to restart the job every once in a while. Attached the EBS read and 
>>> > write metrics. Has anyone encountered this issue and what could be the 
>>> > possible solution.
>>> > 
>>> > We have also tried setting the below rocksdb options but didn't help.
>>> > 
>>> > DBOptions:
>>> > currentOptions.setOptimizeFiltersForHits(true)
>>> > .setWriteBufferSize(536870912)
>>> > .setMaxWriteBufferNumber(5)
>>> > .setMinWriteBufferNumberToMerge(2);
>>> > ColumnFamilyOptions:
>>> > 
>>> > currentOptions.setMaxBackgroundCompactions(4)
>>> > .setMaxManifestFileSize(1048576)
>>> > .setMaxLogFileSize(1048576);
>>> > 
>>> > 
>>> > 
>>> > Thanks.
>>> >  
>>> >  
>>> >  
>>> 
>>> 
>> 
>

Re: [VOTE] Release 1.5.0, release candidate #6

2018-05-25 Thread Aljoscha Krettek 
+1

 - verified signature and hashes
 - ran an example on a kerberized Hadoop 2.8.3 cluster, both with kinit and by 
using keytab

> On 25. May 2018, at 11:56, Piotr Nowojski  wrote:
> 
> Mild +1 from me. I’m concerned about the rate of bugs that we are discovering 
> for each RC, which suggests that there are still some release blockers out 
> there (but I’m not aware of any right now).
> 
> Piotrek
> 
>> On 25 May 2018, at 11:25, Till Rohrmann  wrote:
>> 
>> +1
>> 
>> - Checked checksums and GPG files
>> - Verified that source archives do not contain any binaries
>> - Built Flink with Hadoop 2.8.1 and Scala 2.11.7 from source release
>> - Verified LICENSE and NOTICE file: The LICENSE file contains unnecessary
>> entries for jline-reader and jline-terminal
>> - Checked licenses of newly added dependencies
>> - Checked pom files
>> - Read README.md
>> - Run manual tests in flink-tests: CheckForbiddenMethodsUsage fails
>> because org.apache.flink.queryablestate.messages.KvStateRequest.serialize()
>> uses getBytes without charset. This is non-blocking because it was
>> introduced with 1.4 or before
>> - Checked builds with SBT and the SBT quickstarts
>> - Executed the run-nightly-tests.sh for 10 hours without failures
>> - Executed Jepsen tests without failures
>> 
>> There is a known problem when enabling SSL encryption which can lead to
>> failures [1]. Since there is a workaround for this problem, I would propose
>> to not block the release on it and fix it with Flink 1.5.1.
>> 
>> [1] https://issues.apache.org/jira/browse/FLINK-9437
>> 
>> Cheers,
>> Till
>> 
>> 
>> On Thu, May 24, 2018 at 6:51 PM, Till Rohrmann  wrote:
>> 
>>> Hi everyone,
>>> 
>>> Please review and vote on the release candidate #6 for the version 1.5.0,
>>> as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>> 
>>> 
>>> The complete staging area is available for your review, which includes:
>>> * JIRA release notes [1],
>>> * the official Apache source release and binary convenience releases to be
>>> deployed to dist.apache.org [2], which are signed with the key with
>>> fingerprint 1F302569A96CFFD5 [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "release-1.5.0-rc6" [5],
>>> * PR to update the community web site [6]
>>> 
>>> Please use this document for coordinating testing efforts: [7]
>>> 
>>> The voting periods ends tomorrow at 5pm CET. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>> 
>>> Thanks,
>>> Your friendly Release Manager
>>> 
>>> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>> projectId=12315522=12341764
>>> [2] http://people.apache.org/~trohrmann/flink-1.5.0-rc6/
>>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>>> [4] https://repository.apache.org/content/repositories/
>>> orgapacheflink-1161/
>>> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=commit;h=
>>> c61b108b8eaa22eac3dc492b3c95c22b4177003f
>>> [6] https://github.com/apache/flink-web/pull/106
>>> [7] https://docs.google.com/document/d/10KDBLzt25Z44pdZBSm8MTeKldc4UT
>>> UAbAynfuVQWOt0/edit?usp=sharing
>>> 
>>> Pro-tip: you can create a settings.xml file with these contents:
>>> 
>>> 
>>> 
>>> flink-1.5.0
>>> 
>>> 
>>> 
>>>   flink-1.5.0
>>>   
>>> 
>>>   flink-1.5.0
>>>   
>>>   https://repository.apache.org/content/repositories/
>>> orgapacheflink-1161/
>>>   
>>> 
>>> 
>>>   archetype
>>>   
>>>   https://repository.apache.org/content/repositories/
>>> orgapacheflink-1161/
>>>   
>>> 
>>>   
>>> 
>>> 
>>> 
>>> 
>>> And reference that in you maven commands via --settings
>>> path/to/settings.xml. This is useful for creating a quickstart based on the
>>> staged release and for building against the staged jars.
>>> 
>>> 
>

[jira] [Created] (FLINK-9439) DispatcherTest#testJobRecovery dead locks

2018-05-25 Thread Piotr Nowojski (JIRA) 
Piotr Nowojski created FLINK-9439:
-

 Summary: DispatcherTest#testJobRecovery dead locks
 Key: FLINK-9439
 URL: https://issues.apache.org/jira/browse/FLINK-9439
 Project: Flink
  Issue Type: Bug
  Components: Distributed Coordination
Affects Versions: 1.5.0
Reporter: Piotr Nowojski


Rarely DispatcherTest#testJobRecovery dead locks on master. Example deadlock in 
my pull request:

[https://api.travis-ci.org/v3/job/383147736/log.txt]

afterwards I have stripped down `flink-runtime` and looped on travis this test 
and it also dead locks on master:

[https://github.com/pnowojski/flink/commits/loop-runtime-master]

(note, that looped versions sometimes also fails with an exception from setUp: 
{{ 

akka.actor.InvalidActorNameException: actor name [dispatcher_testJobRecovery] 
is not unique! }} but this might be unrelated).

Example failed build:

[https://travis-ci.org/pnowojski/flink/builds/383650106]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Enable GitBox integration (#2)

2018-05-25 Thread Till Rohrmann 
+1

On Wed, May 23, 2018 at 2:33 PM, Stefan Richter  wrote:

> +1
>
> > Am 23.05.2018 um 14:31 schrieb Stephan Ewen :
> >
> > +1
> >
> > On Wed, May 23, 2018 at 11:11 AM, Fabian Hueske 
> wrote:
> >
> >> +1
> >>
> >> 2018-05-23 8:49 GMT+02:00 Aljoscha Krettek :
> >>
> >>> +1
> >>>
>  On 22. May 2018, at 15:36, Thomas Weise  wrote:
> 
>  +1
> 
> 
>  On Tue, May 22, 2018 at 2:37 AM, Timo Walther 
> >>> wrote:
> 
> > +1
> >
> > Am 22.05.18 um 10:49 schrieb Ted Yu:
> >
> > +1
> >>  Original message From: Chesnay Schepler <
> >> ches...@apache.org> Date: 5/22/18  1:12 AM  (GMT-08:00) To:
> >> dev@flink.apache.org Subject: [VOTE] Enable GitBox integration (#2)
> >> Hello,
> >>
> >> since no concerns have been raised in the discussion about enabling
> >> GitBox [1] I'm opening this vote to make things official.
> >>
> >> Please vote on enabling GitBox integration for the flink and
> >> flink-web
> >> repositories as follows:
> >>
> >> [ ] +1, Approve GitBox
> >> [ ] -1, Do not approve GitBox (please provide specific comments)
> >>
> >> The vote will be open for at least 72 hours. It is adopted by
> >> majority
> >> approval, with at least 3 PMC affirmative votes.
> >>
> >> This is the second attempt for this vote. The previous vote was
> >> cancelled due to the flink-web repository settings. Changes to the
> >> previous vote are highlighted in bold.
> >>
> >> If accepted I will file a ticket with INFRA to enable GitBox with
> the
> >> following settings:
> >>
> >> flink:
> >>
> >>   * no wiki
> >>   * no issues
> >>   * no projects
> >>   * no merge commit button (rebase/squash merge button still
> >> enabled)
> >>   * protected branches [2]: master, release-1.[0-5] (the latter we
> >>> will
> >> have to update with each release)
> >>
> >> flink-web
> >>
> >>   * no wiki
> >>   **no issues*
> >>   * no projects
> >>   * no merge commit button (rebase/squash merge button still
> >> enabled)
> >>   * protected branch: asf-site
> >>   * default branch: asf-site (while we're at it...)
> >>
> >> [1]
> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.
> >> com/DISCUSS-GitBox-td22328.html
> >> [2] https://help.github.com/articles/about-protected-branches/
> >>
> >>
> >>
> >
> >>>
> >>>
> >>
>
>

Re: Merging PR to release branches during RC voting

2018-05-25 Thread Till Rohrmann 
I second Piotr's worries concerning the merging of commits in between RCs.
I think it is a good idea to create a dedicated release-x.y.z branch which
is the basis for RCs for version x.y.z and that all commits being committed
to this branch will go through the RM. This will make him aware of the
included changes between RCs so that he can reschedule some of the testing
tasks if necessary.

If the community agrees on this approach, then we should update the release
guide [1] and the release scripts [2] accordingly. From my side +1 for it.

[1]
https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
[2] https://github.com/apache/flink/tree/master/tools/releasing

Cheers,
Till

On Fri, May 25, 2018 at 11:35 AM, Piotr Nowojski 
wrote:

> Just one clarification: I’m not that much concerned about merging commits
> after feature freeze (especially if they are bug fixes), as merging them
> after release testing kicks of (and even more if they are merged between
> different RC’s). I think in 1.5.0 case, there was a quite large distinction
> between those two dates, but I might be mistaken in this regard.
>
> I agree that having separate branches for minor releases would be cleaner
> and I would be willing to try it out.
>
> Piotrek
>
> > On 25 May 2018, at 04:36, Tzu-Li (Gordon) Tai 
> wrote:
> >
> > Hi Piotr,
> >
> > Thanks for bringing the issue up. I agree that this is something we
> should look into improving.
> >
> > Strictly speaking, I don’t think it is an issue only limited to after a
> RC vote has started.
> > Taking 1.5.0 as an example, release-1.5 was branched out at the
> announcement for feature freeze for nearly a month before the first RC for
> 1.5.0 was created.
> > That period of time was meant for release testing before the actual
> votes took place, but since commits still went through to release-1.5
> during this time without much restrictions, there never really was a
> point-in-time where individual testing efforts could have a commonly agreed
> scope.
> > We usually create RC0s for the initial testing efforts, but by the time
> RC1 is out, it is hard to say that test efforts on RC0 were still valid
> because there was little restraint on what was merged in between.
> >
> > As for the issue under the context of during  RC voting:
> > Up to now, the default way we have mostly used to deal with non-blocking
> bug fixes was to “not block RC votes on them, merge them to the release
> branch and see if it happens to get in if the previous RC was cancelled.”
> > Sometimes, some critical bug fixes were made non-blocking due to the
> fact that it was introduced already in a previous released version.
> > The fact that these critical bug fixes sneak into the release in the
> end, as you mentioned, arguably changes the scope of the release across RCs.
> > Previous RC voters usually expect that previous test efforts can be
> carried over to the new RC because additional commits to the new RC
> *should* only incrementally affect the scope.
> > As the case for the two bug fixes you pointed out (FLINK-9349 and
> FLINK-9295), it is arguable that isn’t the case, since a lot of our
> previous test efforts involved Kafka substantially.
> >
> >> Minor thing is that Jira issues should be updated accordingly. If you
> merge something to release branch, please pay attention to cancelled RCs
> and from which commit new RC is being created.
> >
> >
> > I don’t think this is as minor as it seems. The fact that this
> correction is a very manual process that occurs at a non-anticipated event
> can easily lead to incorrect release notes (which I think has happened a
> few times already in the past, partially because of this issue).
> >
> >> I would propose that any merges to release branches during RC votes
> (after release testing kicked off) should go through release manager, so
> that he could decide whether to include such fixes or postpone them to next
> release. Maybe this would require to create separate release branches for
> 1.5.0 and 1.5.1.
> >
> >
> > Would +1 to this. Just to rephrase this concretely by taking 1.5.0 as an
> example:
> > A specific release-1.5.0 release branch is created from release-1.5 as
> soon as feature freeze is announced.
> > All JIRAs resolved from this point on should have ‘fix version’ set as
> ‘1.5.1’, and commits only go to the release-1.5 branch. They will only be
> included in release-1.5.1 once that branches out to prepare for 1.5.1
> release.
> > Any commits that wish to be included in release-1.5.0 strictly needs to
> go through the release manager; a commit that does go through to
> release-1.5.0 should mean that a new RC should be created and current
> ongoing votes are cancelled.
> > Managing changing such JIRAs’ fix version from ‘1.5.1’ to ‘1.5.0’ should
> be much more easier, since that should happen along with the action of
> bringing over commits to release-1.5.0, and not at the event of an RC being
>

Re: [VOTE] Release 1.5.0, release candidate #6

2018-05-25 Thread Piotr Nowojski 
Mild +1 from me. I’m concerned about the rate of bugs that we are discovering 
for each RC, which suggests that there are still some release blockers out 
there (but I’m not aware of any right now).

Piotrek

> On 25 May 2018, at 11:25, Till Rohrmann  wrote:
> 
> +1
> 
> - Checked checksums and GPG files
> - Verified that source archives do not contain any binaries
> - Built Flink with Hadoop 2.8.1 and Scala 2.11.7 from source release
> - Verified LICENSE and NOTICE file: The LICENSE file contains unnecessary
> entries for jline-reader and jline-terminal
> - Checked licenses of newly added dependencies
> - Checked pom files
> - Read README.md
> - Run manual tests in flink-tests: CheckForbiddenMethodsUsage fails
> because org.apache.flink.queryablestate.messages.KvStateRequest.serialize()
> uses getBytes without charset. This is non-blocking because it was
> introduced with 1.4 or before
> - Checked builds with SBT and the SBT quickstarts
> - Executed the run-nightly-tests.sh for 10 hours without failures
> - Executed Jepsen tests without failures
> 
> There is a known problem when enabling SSL encryption which can lead to
> failures [1]. Since there is a workaround for this problem, I would propose
> to not block the release on it and fix it with Flink 1.5.1.
> 
> [1] https://issues.apache.org/jira/browse/FLINK-9437
> 
> Cheers,
> Till
> 
> 
> On Thu, May 24, 2018 at 6:51 PM, Till Rohrmann  wrote:
> 
>> Hi everyone,
>> 
>> Please review and vote on the release candidate #6 for the version 1.5.0,
>> as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>> 
>> 
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release and binary convenience releases to be
>> deployed to dist.apache.org [2], which are signed with the key with
>> fingerprint 1F302569A96CFFD5 [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "release-1.5.0-rc6" [5],
>> * PR to update the community web site [6]
>> 
>> Please use this document for coordinating testing efforts: [7]
>> 
>> The voting periods ends tomorrow at 5pm CET. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>> 
>> Thanks,
>> Your friendly Release Manager
>> 
>> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12315522=12341764
>> [2] http://people.apache.org/~trohrmann/flink-1.5.0-rc6/
>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [4] https://repository.apache.org/content/repositories/
>> orgapacheflink-1161/
>> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=commit;h=
>> c61b108b8eaa22eac3dc492b3c95c22b4177003f
>> [6] https://github.com/apache/flink-web/pull/106
>> [7] https://docs.google.com/document/d/10KDBLzt25Z44pdZBSm8MTeKldc4UT
>> UAbAynfuVQWOt0/edit?usp=sharing
>> 
>> Pro-tip: you can create a settings.xml file with these contents:
>> 
>> 
>> 
>>  flink-1.5.0
>> 
>> 
>>  
>>flink-1.5.0
>>
>>  
>>flink-1.5.0
>>
>>https://repository.apache.org/content/repositories/
>> orgapacheflink-1161/
>>
>>  
>>  
>>archetype
>>
>>https://repository.apache.org/content/repositories/
>> orgapacheflink-1161/
>>
>>  
>>
>>  
>> 
>> 
>> 
>> And reference that in you maven commands via --settings
>> path/to/settings.xml. This is useful for creating a quickstart based on the
>> staged release and for building against the staged jars.
>> 
>>

Re: [VOTE] Release 1.5.0, release candidate #6

2018-05-25 Thread Till Rohrmann 
+1

- Checked checksums and GPG files
- Verified that source archives do not contain any binaries
- Built Flink with Hadoop 2.8.1 and Scala 2.11.7 from source release
- Verified LICENSE and NOTICE file: The LICENSE file contains unnecessary
entries for jline-reader and jline-terminal
- Checked licenses of newly added dependencies
- Checked pom files
- Read README.md
- Run manual tests in flink-tests: CheckForbiddenMethodsUsage fails
because org.apache.flink.queryablestate.messages.KvStateRequest.serialize()
uses getBytes without charset. This is non-blocking because it was
introduced with 1.4 or before
- Checked builds with SBT and the SBT quickstarts
- Executed the run-nightly-tests.sh for 10 hours without failures
- Executed Jepsen tests without failures

There is a known problem when enabling SSL encryption which can lead to
failures [1]. Since there is a workaround for this problem, I would propose
to not block the release on it and fix it with Flink 1.5.1.

[1] https://issues.apache.org/jira/browse/FLINK-9437

Cheers,
Till


On Thu, May 24, 2018 at 6:51 PM, Till Rohrmann  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #6 for the version 1.5.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint 1F302569A96CFFD5 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-1.5.0-rc6" [5],
> * PR to update the community web site [6]
>
> Please use this document for coordinating testing efforts: [7]
>
> The voting periods ends tomorrow at 5pm CET. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Your friendly Release Manager
>
> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12315522=12341764
> [2] http://people.apache.org/~trohrmann/flink-1.5.0-rc6/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/
> orgapacheflink-1161/
> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=commit;h=
> c61b108b8eaa22eac3dc492b3c95c22b4177003f
> [6] https://github.com/apache/flink-web/pull/106
> [7] https://docs.google.com/document/d/10KDBLzt25Z44pdZBSm8MTeKldc4UT
> UAbAynfuVQWOt0/edit?usp=sharing
>
> Pro-tip: you can create a settings.xml file with these contents:
>
> 
> 
>   flink-1.5.0
> 
> 
>   
> flink-1.5.0
> 
>   
> flink-1.5.0
> 
> https://repository.apache.org/content/repositories/
> orgapacheflink-1161/
> 
>   
>   
> archetype
> 
> https://repository.apache.org/content/repositories/
> orgapacheflink-1161/
> 
>   
> 
>   
> 
> 
>
> And reference that in you maven commands via --settings
> path/to/settings.xml. This is useful for creating a quickstart based on the
> staged release and for building against the staged jars.
>
>

Re: Flink 1.5 - Job fails to execute in multiple taskmanagers (parallelism > 1)

2018-05-25 Thread Edward Rojas 
I just tested the workaround and it works.
Thank you



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

[jira] [Created] (FLINK-9438) Add documentation for (Registry)AvroDeserializationSchema

2018-05-25 Thread Dawid Wysakowicz (JIRA) 
Dawid Wysakowicz created FLINK-9438:
---

 Summary: Add documentation for (Registry)AvroDeserializationSchema
 Key: FLINK-9438
 URL: https://issues.apache.org/jira/browse/FLINK-9438
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Flink 1.5 - Job fails to execute in multiple taskmanagers (parallelism > 1)

2018-05-25 Thread Till Rohrmann 
FYI: https://issues.apache.org/jira/browse/FLINK-9437

We will revert the changes of FLINK-9310 with Flink 1.5.1. For Flink 1.6,
the problem should not arise since we are currently upgrading our Netty
dependency.

Cheers,
Till

On Fri, May 25, 2018 at 8:40 AM, Till Rohrmann  wrote:

> Thanks for reporting the issue Edward.
>
> Taking a look at Netty SslHandler, it looks that we introduced this
> problem with the update of the cipher algorithms [1]. Apparently, the
> SslHandler wants to use inbound heap byte buffer when using a cipher suite
> with GCM enabled [2]. This seems to be fixed with a later version of Netty
> 4 (we are using 4.0.27 at the moment). The problem with the heap byte
> buffers are, that our NettyBufferPool does not support the allocation of
> heap byte buffers in order to have the memory consumption under control.
>
> As a work around, you could set `security.ssl.algorithms` to
> `TLS_RSA_WITH_AES_128_CBC_SHA` in the Flink configuration. That should make
> it work again at the cost of using a cipher which is no longer recommended.
>
> [1] https://issues.apache.org/jira/browse/FLINK-9310.
> [2] https://github.com/netty/netty/blob/netty-4.0.27.Final/
> handler/src/main/java/io/netty/handler/ssl/SslHandler.java#L1218
>
> Cheers,
> Till
>
> On Thu, May 24, 2018 at 8:49 PM, Edward Rojas 
> wrote:
>
>> This may help to target the issue:
>> If I let global ssl enabled, but I set taskmanager.data.ssl.enabled:
>> false,
>> it works again.
>>
>>
>>
>> --
>> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.
>> com/
>>
>
>

[jira] [Created] (FLINK-9437) Revert cypher suite update

2018-05-25 Thread Till Rohrmann (JIRA) 
Till Rohrmann created FLINK-9437:


 Summary: Revert cypher suite update
 Key: FLINK-9437
 URL: https://issues.apache.org/jira/browse/FLINK-9437
 Project: Flink
  Issue Type: Bug
  Components: Security
Affects Versions: 1.5.0
Reporter: Till Rohrmann
 Fix For: 1.5.1


The changes of FLINK-9310 causes Flink to fail when sending data between 
{{TaskManagers}} as reported by a user [1]. The problem seems to be that 
Netty's {{SslHandler}} (v4.0.27) tries to allocate heap buffers when using a 
GCM enabled cypher suite. However, since we explicitly prohibit the allocation 
of heap buffers it fails. In later Netty versions, this behaviour seems to be 
fixed.

I propose to revert the changes of FLINK-9310 and set the default cypher 
algorithm to {{TLS_RSA_WITH_AES_128_CBC_SHA}}.

[1] 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-1-5-Job-fails-to-execute-in-multiple-taskmanagers-parallelism-gt-1-td22467.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Increasing Disk Read Throughput and IOPS

2018-05-25 Thread Stefan Richter 
One more thing, I am aware of one older thread that might be interesting for 
you about RocksDB backend and EBS: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/checkpoint-stuck-with-rocksdb-statebackend-and-s3-filesystem-td18679.html
 


> Am 25.05.2018 um 09:59 schrieb Stefan Richter :
> 
> Hi,
> 
> if the problem is seemingly from reads, I think incremental checkpoints are 
> less likely to cause the problem. What Flink version are you using? Since you 
> mentioned the use of map state, what comes to my mind as a potential cause is 
> described in this issue https://issues.apache.org/jira/browse/FLINK-8639 
>  . This was improved 
> recently. Does the problem also exist for jobs without map state?
> 
> Best,
> Stefan
> 
>> Am 24.05.2018 um 20:25 schrieb Stephan Ewen > >:
>> 
>> One thing that you can always to is disable fsync, because Flink does not 
>> rely on RocksDBs fsync for persistence.
>> 
>> If you disable incremental checkpoints, does that help?
>> If yes, it could be an issue with too many small SSTable files due to 
>> incremental checkpoints (an issue we have on the roadmap to fix).
>> 
>> On Thu, May 24, 2018 at 3:52 PM, Piotr Nowojski > > wrote:
>> Hi,
>> 
>> This issue might have something to do with compaction. Problems with 
>> compaction can especially degrade reads performance (or just increase reads 
>> IO). Have you tried to further enforce more compactions or change 
>> CompactionStyle?
>> 
>> Have you taken a look on 
>> org.apache.flink.contrib.streaming.state.PredefinedOptions?
>> 
>> Maybe Stefan or Andrey could share more input on this.
>> 
>> Piotrek
>> 
>> 
>> > On 22 May 2018, at 08:12, Govindarajan Srinivasaraghavan 
>> > > wrote:
>> > 
>> > Hi All,
>> > 
>> > We are running flink in AWS and we are observing a strange behavior. We 
>> > are using docker containers, EBS for storage and Rocks DB state backend. 
>> > We have a few map and value states with checkpointing every 30 seconds and 
>> > incremental checkpointing turned on. The issue we are noticing is the read 
>> > IOPS and read throughput gradually increases over time and keeps 
>> > constantly growing. The write throughput and write bytes are not 
>> > increasing as much as reads. The checkpoints are written to a durable NFS 
>> > storage. We are not sure what is causing this constant increase in read 
>> > throughput but due to which we are running out of EBS burst balance and 
>> > need to restart the job every once in a while. Attached the EBS read and 
>> > write metrics. Has anyone encountered this issue and what could be the 
>> > possible solution.
>> > 
>> > We have also tried setting the below rocksdb options but didn't help.
>> > 
>> > DBOptions:
>> > currentOptions.setOptimizeFiltersForHits(true)
>> > .setWriteBufferSize(536870912)
>> > .setMaxWriteBufferNumber(5)
>> > .setMinWriteBufferNumberToMerge(2);
>> > ColumnFamilyOptions:
>> > 
>> > currentOptions.setMaxBackgroundCompactions(4)
>> > .setMaxManifestFileSize(1048576)
>> > .setMaxLogFileSize(1048576);
>> > 
>> > 
>> > 
>> > Thanks.
>> >  
>> >  
>> >  
>> 
>> 
>

[jira] [Created] (FLINK-9436) Remove generic parameter namespace from InternalTimeServiceManager

2018-05-25 Thread Stefan Richter (JIRA) 
Stefan Richter created FLINK-9436:
-

 Summary: Remove generic parameter namespace from 
InternalTimeServiceManager
 Key: FLINK-9436
 URL: https://issues.apache.org/jira/browse/FLINK-9436
 Project: Flink
  Issue Type: Improvement
  Components: State Backends, Checkpointing
Affects Versions: 1.6.0
Reporter: Stefan Richter
Assignee: Stefan Richter
 Fix For: 1.6.0


Currently {{InternalTimeServiceManager}} declares a generic parameter 
{{N}} for namespace. This parameter is not useful or even misleading, because 
in the current implementation the managed timer services can all have different 
types for their {{N}}. I suggest to remove the parameter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Increasing Disk Read Throughput and IOPS

2018-05-25 Thread Stefan Richter 
Hi,

if the problem is seemingly from reads, I think incremental checkpoints are 
less likely to cause the problem. What Flink version are you using? Since you 
mentioned the use of map state, what comes to my mind as a potential cause is 
described in this issue https://issues.apache.org/jira/browse/FLINK-8639 
 . This was improved 
recently. Does the problem also exist for jobs without map state?

Best,
Stefan

> Am 24.05.2018 um 20:25 schrieb Stephan Ewen :
> 
> One thing that you can always to is disable fsync, because Flink does not 
> rely on RocksDBs fsync for persistence.
> 
> If you disable incremental checkpoints, does that help?
> If yes, it could be an issue with too many small SSTable files due to 
> incremental checkpoints (an issue we have on the roadmap to fix).
> 
> On Thu, May 24, 2018 at 3:52 PM, Piotr Nowojski  > wrote:
> Hi,
> 
> This issue might have something to do with compaction. Problems with 
> compaction can especially degrade reads performance (or just increase reads 
> IO). Have you tried to further enforce more compactions or change 
> CompactionStyle?
> 
> Have you taken a look on 
> org.apache.flink.contrib.streaming.state.PredefinedOptions?
> 
> Maybe Stefan or Andrey could share more input on this.
> 
> Piotrek
> 
> 
> > On 22 May 2018, at 08:12, Govindarajan Srinivasaraghavan 
> > > wrote:
> > 
> > Hi All,
> > 
> > We are running flink in AWS and we are observing a strange behavior. We are 
> > using docker containers, EBS for storage and Rocks DB state backend. We 
> > have a few map and value states with checkpointing every 30 seconds and 
> > incremental checkpointing turned on. The issue we are noticing is the read 
> > IOPS and read throughput gradually increases over time and keeps constantly 
> > growing. The write throughput and write bytes are not increasing as much as 
> > reads. The checkpoints are written to a durable NFS storage. We are not 
> > sure what is causing this constant increase in read throughput but due to 
> > which we are running out of EBS burst balance and need to restart the job 
> > every once in a while. Attached the EBS read and write metrics. Has anyone 
> > encountered this issue and what could be the possible solution.
> > 
> > We have also tried setting the below rocksdb options but didn't help.
> > 
> > DBOptions:
> > currentOptions.setOptimizeFiltersForHits(true)
> > .setWriteBufferSize(536870912)
> > .setMaxWriteBufferNumber(5)
> > .setMinWriteBufferNumberToMerge(2);
> > ColumnFamilyOptions:
> > 
> > currentOptions.setMaxBackgroundCompactions(4)
> > .setMaxManifestFileSize(1048576)
> > .setMaxLogFileSize(1048576);
> > 
> > 
> > 
> > Thanks.
> >  
> >  
> >  
> 
>

Re: Flink 1.5 - Job fails to execute in multiple taskmanagers (parallelism > 1)

2018-05-25 Thread Till Rohrmann 
Thanks for reporting the issue Edward.

Taking a look at Netty SslHandler, it looks that we introduced this problem
with the update of the cipher algorithms [1]. Apparently, the SslHandler
wants to use inbound heap byte buffer when using a cipher suite with GCM
enabled [2]. This seems to be fixed with a later version of Netty 4 (we are
using 4.0.27 at the moment). The problem with the heap byte buffers are,
that our NettyBufferPool does not support the allocation of heap byte
buffers in order to have the memory consumption under control.

As a work around, you could set `security.ssl.algorithms` to
`TLS_RSA_WITH_AES_128_CBC_SHA` in the Flink configuration. That should make
it work again at the cost of using a cipher which is no longer recommended.

[1] https://issues.apache.org/jira/browse/FLINK-9310.
[2]
https://github.com/netty/netty/blob/netty-4.0.27.Final/handler/src/main/java/io/netty/handler/ssl/SslHandler.java#L1218

Cheers,
Till

On Thu, May 24, 2018 at 8:49 PM, Edward Rojas 
wrote:

> This may help to target the issue:
> If I let global ssl enabled, but I set taskmanager.data.ssl.enabled: false,
> it works again.
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>