Re: [ANNOUNCE] New committer Gary Yao

[Rong Rong]

Congratulations Gary!!

On Fri, Sep 7, 2018 at 7:38 PM bupt_ljy  wrote:

> Congratulations Gary!
>
>
> Jiayi
>
>
> Original Message
> Sender:vino yangyanghua1...@gmail.com
> Recipient:dev...@flink.apache.org
> Date:Saturday, Sep 8, 2018 10:11
> Subject:Re: [ANNOUNCE] New committer Gary Yao
>
>
> Congratulations Gary! Chen Qin qinnc...@gmail.com 于2018年9月8日周六 上午2:07写道：
> Congrats!   ChenOn Sep 7, 2018, at 10:51, Xingcan Cui
> xingc...@gmail.com wrote: Congratulations, Gary! Xingcan On
> Sep 7, 2018, at 11:20 PM, Hequn Cheng chenghe...@gmail.com wrote:
>  Congratulations Gary! Hequn On Fri, Sep 7, 2018 at 11:16 PM
> Matthias J. Sax mj...@apache.org  wrote: Congrats! On 09/07/2018
> 08:15 AM, Timo Walther wrote:   Congratulations, Gary! Timo   Am
> 07.09.18 um 16:46 schrieb Ufuk Celebi:   Great addition to the committers.
> Congrats, Gary! – Ufuk   On Fri, Sep 7, 2018 at 4:45 PM, Kostas
> Kloudas   k.klou...@data-artisans.com wrote:   Congratulations Gary! Well
> deserved! Cheers,   Kostas On Sep 7, 2018, at 4:43 PM, Fabian
> Hueske fhue...@gmail.com  wrote: Congratulations Gary! 2018-09-07
> 16:29 GMT+02:00 Thomas Weise t...@apache.org: Congrats, Gary! On
> Fri, Sep 7, 2018 at 4:17 PM Dawid Wysakowicz   dwysakow...@apache.org
>  wrote: Congratulations Gary! Well deserved! On 07/09/18 16:00,
> zhangmingleihe wrote:   Congrats Gary! Cheers   Minglei 在
> 2018年9月7日，下午9:59，Andrey Zagrebin   and...@data-artisans.com 写道：
>  Congratulations Gary! On 7 Sep 2018, at 15:45, Stefan Richter
> s.rich...@data-artisans.com   wrote:   Congrats Gary! Am 07.09.2018
> um 15:14 schrieb Till Rohrmann   trohrm...@apache.org   :   Hi
> everybody, On behalf of the PMC I am delighted to announce Gary Yao as
> a   new   Flink   committer! Gary started contributing to the project
> in June 2017. He  helped   with   the   Flip-6 implementation, implemented
> many of the new REST   handlers,   fixed   Mesos issues and initiated the
> Jepsen-based distributed test   suite   which   uncovered several serious
> issues. Moreover, he actively helps   community   members on the mailing
> list and with PR reviews. Please join me in congratulating Gary for
> becoming a Flink   committer!   Cheers,   Till

Re: [ANNOUNCE] New committer Gary Yao

[bupt_ljy]

Congratulations Gary!


Jiayi


Original Message
Sender:vino yangyanghua1...@gmail.com
Recipient:dev...@flink.apache.org
Date:Saturday, Sep 8, 2018 10:11
Subject:Re: [ANNOUNCE] New committer Gary Yao


Congratulations Gary! Chen Qin qinnc...@gmail.com 于2018年9月8日周六 上午2:07写道：  
Congrats!   ChenOn Sep 7, 2018, at 10:51, Xingcan Cui xingc...@gmail.com 
wrote: Congratulations, Gary! Xingcan On Sep 7, 2018, at 11:20 PM, 
Hequn Cheng chenghe...@gmail.com wrote: Congratulations Gary! Hequn 
On Fri, Sep 7, 2018 at 11:16 PM Matthias J. Sax mj...@apache.org  wrote: 
Congrats! On 09/07/2018 08:15 AM, Timo Walther wrote:   Congratulations, 
Gary! Timo   Am 07.09.18 um 16:46 schrieb Ufuk Celebi:   Great addition 
to the committers. Congrats, Gary! – Ufuk   On Fri, Sep 7, 2018 at 4:45 
PM, Kostas Kloudas   k.klou...@data-artisans.com wrote:   Congratulations Gary! 
Well deserved! Cheers,   Kostas On Sep 7, 2018, at 4:43 PM, Fabian 
Hueske fhue...@gmail.com  wrote: Congratulations Gary! 2018-09-07 16:29 
GMT+02:00 Thomas Weise t...@apache.org: Congrats, Gary! On Fri, Sep 7, 
2018 at 4:17 PM Dawid Wysakowicz   dwysakow...@apache.org   wrote: 
Congratulations Gary! Well deserved! On 07/09/18 16:00, zhangmingleihe 
wrote:   Congrats Gary! Cheers   Minglei 在 2018年9月7日，下午9:59，Andrey 
Zagrebin   and...@data-artisans.com 写道： Congratulations Gary! On 7 Sep 
2018, at 15:45, Stefan Richter   s.rich...@data-artisans.com   wrote:   
Congrats Gary! Am 07.09.2018 um 15:14 schrieb Till Rohrmann   
trohrm...@apache.org   :   Hi everybody, On behalf of the PMC I am 
delighted to announce Gary Yao as a   new   Flink   committer! Gary started 
contributing to the project in June 2017. He  helped   with   the   Flip-6 
implementation, implemented many of the new REST   handlers,   fixed   Mesos 
issues and initiated the Jepsen-based distributed test   suite   which   
uncovered several serious issues. Moreover, he actively helps   community   
members on the mailing list and with PR reviews. Please join me in 
congratulating Gary for becoming a Flink   committer!   Cheers,   Till

Re: [ANNOUNCE] New committer Gary Yao

[vino yang]

Congratulations Gary!

Chen Qin  于2018年9月8日周六 上午2:07写道：

> Congrats!
>
> Chen
>
> > On Sep 7, 2018, at 10:51, Xingcan Cui  wrote:
> >
> > Congratulations, Gary!
> >
> > Xingcan
> >
> >> On Sep 7, 2018, at 11:20 PM, Hequn Cheng  wrote:
> >>
> >> Congratulations Gary!
> >>
> >> Hequn
> >>
> >>> On Fri, Sep 7, 2018 at 11:16 PM Matthias J. Sax 
> wrote:
> >>>
> >>> Congrats!
> >>>
>  On 09/07/2018 08:15 AM, Timo Walther wrote:
>  Congratulations, Gary!
> 
>  Timo
> 
> 
> > Am 07.09.18 um 16:46 schrieb Ufuk Celebi:
> > Great addition to the committers. Congrats, Gary!
> >
> > – Ufuk
> >
> >
> > On Fri, Sep 7, 2018 at 4:45 PM, Kostas Kloudas
> >  wrote:
> >> Congratulations Gary! Well deserved!
> >>
> >> Cheers,
> >> Kostas
> >>
> >>> On Sep 7, 2018, at 4:43 PM, Fabian Hueske 
> wrote:
> >>>
> >>> Congratulations Gary!
> >>>
> >>> 2018-09-07 16:29 GMT+02:00 Thomas Weise :
> >>>
>  Congrats, Gary!
> 
>  On Fri, Sep 7, 2018 at 4:17 PM Dawid Wysakowicz
>  
>  wrote:
> 
> > Congratulations Gary! Well deserved!
> >
> >> On 07/09/18 16:00, zhangmingleihe wrote:
> >> Congrats Gary!
> >>
> >> Cheers
> >> Minglei
> >>
> >>> 在 2018年9月7日，下午9:59，Andrey Zagrebin
> >>>  写道：
> >>>
> >>> Congratulations Gary!
> >>>
>  On 7 Sep 2018, at 15:45, Stefan Richter
>   > wrote:
>  Congrats Gary!
> 
> > Am 07.09.2018 um 15:14 schrieb Till Rohrmann
> >  > :
> > Hi everybody,
> >
> > On behalf of the PMC I am delighted to announce Gary Yao as a
> >>> new
> > Flink
> > committer!
> >
> > Gary started contributing to the project in June 2017. He
> helped
>  with
> > the
> > Flip-6 implementation, implemented many of the new REST
> >>> handlers,
> > fixed
> > Mesos issues and initiated the Jepsen-based distributed test
> > suite
> > which
> > uncovered several serious issues. Moreover, he actively helps
> > community
> > members on the mailing list and with PR reviews.
> >
> > Please join me in congratulating Gary for becoming a Flink
>  committer!
> > Cheers,
> > Till
> >
> 
> >>>
> >>>
> >
>

[jira] [Created] (FLINK-10305) flink-conf.yaml grows continuously

[Dimitrije (JIRA)]

Dimitrije created FLINK-10305:
-

 Summary: flink-conf.yaml grows continuously  
 Key: FLINK-10305
 URL: https://issues.apache.org/jira/browse/FLINK-10305
 Project: Flink
  Issue Type: Bug
Reporter: Dimitrije


`query.server.port` & `blob.server.port` variables are continuously appended to 
the flink-conf.yaml when the job manager restarts.

 

Running a jobmanager & taskmanager using docker-compose

I am using a single `flink-conf.yaml` which is mounted as a volume to a 
jobmanager and taskmanager container.  Every time the jobmanager restarts, it 
appends the two variables to the end of the file causing it to grow.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New committer Gary Yao

[Chen Qin]

Congrats!

Chen

> On Sep 7, 2018, at 10:51, Xingcan Cui  wrote:
> 
> Congratulations, Gary!
> 
> Xingcan
> 
>> On Sep 7, 2018, at 11:20 PM, Hequn Cheng  wrote:
>> 
>> Congratulations Gary!
>> 
>> Hequn
>> 
>>> On Fri, Sep 7, 2018 at 11:16 PM Matthias J. Sax  wrote:
>>> 
>>> Congrats!
>>> 
 On 09/07/2018 08:15 AM, Timo Walther wrote:
 Congratulations, Gary!
 
 Timo
 
 
> Am 07.09.18 um 16:46 schrieb Ufuk Celebi:
> Great addition to the committers. Congrats, Gary!
> 
> – Ufuk
> 
> 
> On Fri, Sep 7, 2018 at 4:45 PM, Kostas Kloudas
>  wrote:
>> Congratulations Gary! Well deserved!
>> 
>> Cheers,
>> Kostas
>> 
>>> On Sep 7, 2018, at 4:43 PM, Fabian Hueske  wrote:
>>> 
>>> Congratulations Gary!
>>> 
>>> 2018-09-07 16:29 GMT+02:00 Thomas Weise :
>>> 
 Congrats, Gary!
 
 On Fri, Sep 7, 2018 at 4:17 PM Dawid Wysakowicz
 
 wrote:
 
> Congratulations Gary! Well deserved!
> 
>> On 07/09/18 16:00, zhangmingleihe wrote:
>> Congrats Gary!
>> 
>> Cheers
>> Minglei
>> 
>>> 在 2018年9月7日，下午9:59，Andrey Zagrebin
>>>  写道：
>>> 
>>> Congratulations Gary!
>>> 
 On 7 Sep 2018, at 15:45, Stefan Richter
  wrote:
 Congrats Gary!
 
> Am 07.09.2018 um 15:14 schrieb Till Rohrmann
>  :
> Hi everybody,
> 
> On behalf of the PMC I am delighted to announce Gary Yao as a
>>> new
> Flink
> committer!
> 
> Gary started contributing to the project in June 2017. He helped
 with
> the
> Flip-6 implementation, implemented many of the new REST
>>> handlers,
> fixed
> Mesos issues and initiated the Jepsen-based distributed test
> suite
> which
> uncovered several serious issues. Moreover, he actively helps
> community
> members on the mailing list and with PR reviews.
> 
> Please join me in congratulating Gary for becoming a Flink
 committer!
> Cheers,
> Till
> 
 
>>> 
>>> 
> 

Re: [ANNOUNCE] New committer Gary Yao

[Xingcan Cui]

Congratulations, Gary!

Xingcan

> On Sep 7, 2018, at 11:20 PM, Hequn Cheng  wrote:
> 
> Congratulations Gary!
> 
> Hequn
> 
> On Fri, Sep 7, 2018 at 11:16 PM Matthias J. Sax  wrote:
> 
>> Congrats!
>> 
>> On 09/07/2018 08:15 AM, Timo Walther wrote:
>>> Congratulations, Gary!
>>> 
>>> Timo
>>> 
>>> 
>>> Am 07.09.18 um 16:46 schrieb Ufuk Celebi:
 Great addition to the committers. Congrats, Gary!
 
 – Ufuk
 
 
 On Fri, Sep 7, 2018 at 4:45 PM, Kostas Kloudas
  wrote:
> Congratulations Gary! Well deserved!
> 
> Cheers,
> Kostas
> 
>> On Sep 7, 2018, at 4:43 PM, Fabian Hueske  wrote:
>> 
>> Congratulations Gary!
>> 
>> 2018-09-07 16:29 GMT+02:00 Thomas Weise :
>> 
>>> Congrats, Gary!
>>> 
>>> On Fri, Sep 7, 2018 at 4:17 PM Dawid Wysakowicz
>>> 
>>> wrote:
>>> 
 Congratulations Gary! Well deserved!
 
 On 07/09/18 16:00, zhangmingleihe wrote:
> Congrats Gary!
> 
> Cheers
> Minglei
> 
>> 在 2018年9月7日，下午9:59，Andrey Zagrebin
>>  写道：
>> 
>> Congratulations Gary!
>> 
>>> On 7 Sep 2018, at 15:45, Stefan Richter
>>> >>> wrote:
>>> Congrats Gary!
>>> 
 Am 07.09.2018 um 15:14 schrieb Till Rohrmann
 >>> :
 Hi everybody,
 
 On behalf of the PMC I am delighted to announce Gary Yao as a
>> new
 Flink
 committer!
 
 Gary started contributing to the project in June 2017. He helped
>>> with
 the
 Flip-6 implementation, implemented many of the new REST
>> handlers,
 fixed
 Mesos issues and initiated the Jepsen-based distributed test
 suite
 which
 uncovered several serious issues. Moreover, he actively helps
 community
 members on the mailing list and with PR reviews.
 
 Please join me in congratulating Gary for becoming a Flink
>>> committer!
 Cheers,
 Till
 
>>> 
>> 
>> 

[jira] [Created] (FLINK-10304) Remove deprecated AbstractYarnClusterDescriptor field

[JIRA]

陈梓立 created FLINK-10304:
---

 Summary: Remove deprecated AbstractYarnClusterDescriptor field
 Key: FLINK-10304
 URL: https://issues.apache.org/jira/browse/FLINK-10304
 Project: Flink
  Issue Type: Improvement
  Components: Client, YARN
Affects Versions: 1.7.0
Reporter: 陈梓立
Assignee: 陈梓立
 Fix For: 1.7.0


Depend on [~trohrm...@apache.org]'s 
[commit|https://github.com/apache/flink/commit/6356128865bff7463bf03185d18b129ed3633bc2],
 {{AbstractYarnClusterDescriptor}} should not care whether it is in DETACHED 
mode.

After digging I found the main usages of it are

1. {{FlinkYarnSessionCli#run}}, this can be resolved by checking whether 
{{allOptions}} has {{DETACHED_OPTION}} locally.

2. when AbstractYarnClusterDescriptor start a AM, it sets 
{{appMasterEnv.put(YarnConfigKeys.ENV_DETACHED, String.valueOf(detached));}}. 
At this point it seems that YarnClusterDescriptor should know whether or not it 
is in detached mode.

If usage 2 is irrelevant now, we can get rid of deprecated method in FLIP-6 
codebase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10303) Fix critical vulnerabilities Python API

[Konstantin Knauf (JIRA)]

Konstantin Knauf created FLINK-10303:


 Summary: Fix critical vulnerabilities Python API
 Key: FLINK-10303
 URL: https://issues.apache.org/jira/browse/FLINK-10303
 Project: Flink
  Issue Type: Improvement
  Components: Python API
Affects Versions: 1.6.0
Reporter: Konstantin Knauf


A user has reported two "critical" vulnerabilities in the Python API, which we 
should probably fix: 

* https://nvd.nist.gov/vuln/detail/CVE-2016-4000
* https://cwe.mitre.org/data/definitions/384.html in 
flink-streaming-python_2.11-1.6.0.jar <= pip-1.6-py2.py3-none-any.whl <= 
sessions.py : [2.1.0, 2.6.0)

For users, who don't need the Python API, an easy work-around is exclude the 
flink-streaming-python_2.11.jar from the distribution. 
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: 答复: Flink1.6.0 submit job and got "No content to map due to end-of-input" Error

[Till Rohrmann]

Hi Gongsen,

Chesnay found and fixed the problem:
https://issues.apache.org/jira/browse/FLINK-10293.

Cheers,
Till


On Wed, Sep 5, 2018 at 10:00 AM 潘 功森  wrote:

> Hi  Chesney,
>
>I can sure the client and cluster all upgraded to 1.6.0 cause if I
> used “./flink run XXX.jar” to submit a job and it works fine.You can see ui
> below.
>
>But when I used createRemoteEnvironment at local, and it failed.It
> confused me a lot.
>
>
>
>
> --
> *发件人:* Chesnay Schepler 
> *发送时间:* Wednesday, September 5, 2018 3:23:23 PM
> *收件人:* 潘 功森; vino yang; dev@flink.apache.org
> *抄送:* user
> *主题:* Re: 答复: Flink1.6.0 submit job and got "No content to map due to
> end-of-input" Error
>
> Did you upgrade both the client and cluster to 1.6.0? The server returned
> a completely empty response which shouldn't be possible if it runs 1.6.0.
>
> On 05.09.2018 07:27, 潘 功森 wrote:
>
> Hi  Vino，
>
>
>
> Below are dependencies I used，please have a look.
>
>
>
> I floud it also inclued flink-connector-kafka-0.10_2.11-1.6.0.jar and
> flink-connector-kafka-0.9_2.11-1.6.0.jar, and I don’t know if it has any
> effect?
>
>
>
> yours,
>
> Gongsen
>
>
>
> 发送自 Windows 10 版邮件 应用
>
>
> --
> *发件人:* vino yang  
> *发送时间:* Wednesday, September 5, 2018 10:35:59 AM
> *收件人:* dev@flink.apache.org
> *抄送:* user
> *主题:* Re: Flink1.6.0 submit job and got "No content to map due to
> end-of-input" Error
>
> Hi Pangongsen,
>
> Do you upgrade the Flink-related dependencies you use at the same time? In
> other words, is the dependency consistent with the flink version?
>
> Thanks, vino.
>
> ? ??  于2018年9月4日周二 下午10:07写道：
>
>> Hi all,
>>  I use below way to submit jar to Flink :
>>
>> StreamExecutionEnvironment env =
>> StreamExecutionEnvironment.createRemoteEnvironment(config.clusterIp,
>>
>>   config.clusterPort,
>>
>>   config.clusterFlinkJar);
>>
>>
>> I used Flink 1.3.2 before, and it works fine. But I upgrade it to
>> 1.6.0, and I got the error below:
>>
>> 2018-09-04 21:38:32.039 [ERROR] [flink-rest-client-netty-19-1]
>> org.apache.flink.runtime.rest.RestClient - Unexpected plain-text response:
>>
>> 2018-09-04 21:38:32.137 [ERROR] [flink-rest-client-netty-18-1]
>> org.apache.flink.runtime.rest.RestClient - Response was not valid JSON.
>>
>> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException:
>> No content to map due to end-of-input
>>
>>
>> Could you give me some advice to fix it?
>>
>> yours,
>> Gongsen
>>
>>
>

Re: Cancel flink job occur exception.

[Till Rohrmann]

Hi Vino and Devin,

could you maybe send us the cluster entrypoint and client logs once you
observe the exception? That way it will be possible to debug it.

Cheers,
Till

On Tue, Sep 4, 2018 at 2:26 PM vino yang  wrote:

> Hi Devin,
>
> Why do you trigger cancel with savepoint immediately after the job status
> changes to Deployed? A more secure way is to wait for the job to become
> running after it has been running for a while before triggering.
>
> We have also encountered before, there will be a case where the client
> times out or still tries to connect to the closed JM after RestClient calls
> cancel with savepoint.
>
> Thanks, vino.
>
> devinduan(段丁瑞)  于2018年9月4日周二 下午6:22写道：
>
> > Hi all,
> >   I submit a flink job through yarn-cluster mode and cancel job with
> > savepoint option immediately after job status change to deployed.
> Sometimes
> > i met this error:
> >
> > org.apache.flink.util.FlinkException: Could not cancel job .
> > at
> >
> org.apache.flink.client.cli.CliFrontend.lambda$cancel$4(CliFrontend.java:585)
> > at
> >
> org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:960)
> > at
> > org.apache.flink.client.cli.CliFrontend.cancel(CliFrontend.java:577)
> > at
> >
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1034)
> > at java.lang.Thread.run(Thread.java:748)
> > Caused by: java.util.concurrent.ExecutionException:
> > org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not
> > complete the operation. Number of retries has been exhausted.
> > at
> >
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> > at
> > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> > at
> >
> org.apache.flink.client.program.rest.RestClusterClient.cancelWithSavepoint(RestClusterClient.java:398)
> > at
> >
> org.apache.flink.client.cli.CliFrontend.lambda$cancel$4(CliFrontend.java:583)
> > ... 6 more
> > Caused by:
> org.apache.flink.runtime.concurrent.FutureUtils$RetryException:
> > Could not complete the operation. Number of retries has been exhausted.
> > at
> >
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$5(FutureUtils.java:213)
> > at
> >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> > at
> >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> > at
> >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> > at
> >
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> > at
> >
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:274)
> > at
> >
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> > at
> >
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> > at
> >
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> > at
> >
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
> > at
> >
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:268)
> > at
> >
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:284)
> > at
> >
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
> > at
> >
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> > at
> >
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> > at
> >
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> > at
> >
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> > at
> >
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> > ... 1 more
> > Caused by: java.util.concurrent.CompletionException:
> > java.net.ConnectException: Connect refuse: xxx/xxx.xxx.xxx.xxx:xxx
> > at
> >
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> > at
> >
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> > at
> >
> java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943)
> > at
> >
> 

Re: Cancel flink job occur exception

[Till Rohrmann]

Hi Devin,

could you send the logs of the cluster entrypoint and the client once you
see this exception? This will help to debug the problem.

Cheers,
Till

On Tue, Sep 4, 2018 at 2:12 PM devinduan(段丁瑞)  wrote:

> Hi all,
>   I submit a flink job through yarn-cluster mode and cancel job with
> savepoint option immediately after job status change to deployed. Sometimes
> i met this error:
>
> org.apache.flink.util.FlinkException: Could not cancel job .
> at
> org.apache.flink.client.cli.CliFrontend.lambda$cancel$4(CliFrontend.java:585)
> at
> org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:960)
> at
> org.apache.flink.client.cli.CliFrontend.cancel(CliFrontend.java:577)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1034)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not
> complete the operation. Number of retries has been exhausted.
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.flink.client.program.rest.RestClusterClient.cancelWithSavepoint(RestClusterClient.java:398)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$cancel$4(CliFrontend.java:583)
> ... 6 more
> Caused by: org.apache.flink.runtime.concurrent.FutureUtils$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$5(FutureUtils.java:213)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> ... 1 more
> Caused by: java.util.concurrent.CompletionException:
> java.net.ConnectException: Connect refuse: xxx/xxx.xxx.xxx.xxx:xxx
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> at
> java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943)
> at
> java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
> ... 16 more
> Caused by: java.net.ConnectException: Connect refuse:
> xxx/xxx.xxx.xxx.xxx:xxx
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:281)
> ... 7 more
>
> I check the jobmanager log, no error found. Savepoint is correct saved
> in hdfs. Yarn appliction status changed to FINISHED and FinalStatus change
> to KILLED.
> I think this issue occur because RestClusterClient cannot find
> jobmanager addresss after Jobmanager(AM) has shutdown.
> My flink version is 1.5.3.
> Anyone could help me to resolve this issue, thanks!
>
> devin.
>
>

Re: [ANNOUNCE] New committer Gary Yao

[Hequn Cheng]

Congratulations Gary!

Hequn

On Fri, Sep 7, 2018 at 11:16 PM Matthias J. Sax  wrote:

> Congrats!
>
> On 09/07/2018 08:15 AM, Timo Walther wrote:
> > Congratulations, Gary!
> >
> > Timo
> >
> >
> > Am 07.09.18 um 16:46 schrieb Ufuk Celebi:
> >> Great addition to the committers. Congrats, Gary!
> >>
> >> – Ufuk
> >>
> >>
> >> On Fri, Sep 7, 2018 at 4:45 PM, Kostas Kloudas
> >>  wrote:
> >>> Congratulations Gary! Well deserved!
> >>>
> >>> Cheers,
> >>> Kostas
> >>>
>  On Sep 7, 2018, at 4:43 PM, Fabian Hueske  wrote:
> 
>  Congratulations Gary!
> 
>  2018-09-07 16:29 GMT+02:00 Thomas Weise :
> 
> > Congrats, Gary!
> >
> > On Fri, Sep 7, 2018 at 4:17 PM Dawid Wysakowicz
> > 
> > wrote:
> >
> >> Congratulations Gary! Well deserved!
> >>
> >> On 07/09/18 16:00, zhangmingleihe wrote:
> >>> Congrats Gary!
> >>>
> >>> Cheers
> >>> Minglei
> >>>
>  在 2018年9月7日，下午9:59，Andrey Zagrebin
>   写道：
> 
>  Congratulations Gary!
> 
> > On 7 Sep 2018, at 15:45, Stefan Richter
> >  >> wrote:
> > Congrats Gary!
> >
> >> Am 07.09.2018 um 15:14 schrieb Till Rohrmann
> >>  >> :
> >> Hi everybody,
> >>
> >> On behalf of the PMC I am delighted to announce Gary Yao as a
> new
> >> Flink
> >> committer!
> >>
> >> Gary started contributing to the project in June 2017. He helped
> > with
> >> the
> >> Flip-6 implementation, implemented many of the new REST
> handlers,
> >> fixed
> >> Mesos issues and initiated the Jepsen-based distributed test
> >> suite
> >> which
> >> uncovered several serious issues. Moreover, he actively helps
> >> community
> >> members on the mailing list and with PR reviews.
> >>
> >> Please join me in congratulating Gary for becoming a Flink
> > committer!
> >> Cheers,
> >> Till
> >>
> >
>
>

Re: [ANNOUNCE] New committer Gary Yao

[Matthias J. Sax]

Congrats!

On 09/07/2018 08:15 AM, Timo Walther wrote:
> Congratulations, Gary!
> 
> Timo
> 
> 
> Am 07.09.18 um 16:46 schrieb Ufuk Celebi:
>> Great addition to the committers. Congrats, Gary!
>>
>> – Ufuk
>>
>>
>> On Fri, Sep 7, 2018 at 4:45 PM, Kostas Kloudas
>>  wrote:
>>> Congratulations Gary! Well deserved!
>>>
>>> Cheers,
>>> Kostas
>>>
 On Sep 7, 2018, at 4:43 PM, Fabian Hueske  wrote:

 Congratulations Gary!

 2018-09-07 16:29 GMT+02:00 Thomas Weise :

> Congrats, Gary!
>
> On Fri, Sep 7, 2018 at 4:17 PM Dawid Wysakowicz
> 
> wrote:
>
>> Congratulations Gary! Well deserved!
>>
>> On 07/09/18 16:00, zhangmingleihe wrote:
>>> Congrats Gary!
>>>
>>> Cheers
>>> Minglei
>>>
 在 2018年9月7日，下午9:59，Andrey Zagrebin
  写道：

 Congratulations Gary!

> On 7 Sep 2018, at 15:45, Stefan Richter
> > wrote:
> Congrats Gary!
>
>> Am 07.09.2018 um 15:14 schrieb Till Rohrmann
>> > :
>> Hi everybody,
>>
>> On behalf of the PMC I am delighted to announce Gary Yao as a new
>> Flink
>> committer!
>>
>> Gary started contributing to the project in June 2017. He helped
> with
>> the
>> Flip-6 implementation, implemented many of the new REST handlers,
>> fixed
>> Mesos issues and initiated the Jepsen-based distributed test
>> suite
>> which
>> uncovered several serious issues. Moreover, he actively helps
>> community
>> members on the mailing list and with PR reviews.
>>
>> Please join me in congratulating Gary for becoming a Flink
> committer!
>> Cheers,
>> Till
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [ANNOUNCE] New committer Gary Yao

[Timo Walther]

Congratulations, Gary!

Timo


Am 07.09.18 um 16:46 schrieb Ufuk Celebi:

Great addition to the committers. Congrats, Gary!

– Ufuk


On Fri, Sep 7, 2018 at 4:45 PM, Kostas Kloudas
 wrote:

Congratulations Gary! Well deserved!

Cheers,
Kostas


On Sep 7, 2018, at 4:43 PM, Fabian Hueske  wrote:

Congratulations Gary!

2018-09-07 16:29 GMT+02:00 Thomas Weise :


Congrats, Gary!

On Fri, Sep 7, 2018 at 4:17 PM Dawid Wysakowicz 
wrote:


Congratulations Gary! Well deserved!

On 07/09/18 16:00, zhangmingleihe wrote:

Congrats Gary!

Cheers
Minglei


在 2018年9月7日，下午9:59，Andrey Zagrebin  写道：

Congratulations Gary!


On 7 Sep 2018, at 15:45, Stefan Richter 
wrote:

Congrats Gary!


Am 07.09.2018 um 15:14 schrieb Till Rohrmann 
:

Hi everybody,

On behalf of the PMC I am delighted to announce Gary Yao as a new

Flink

committer!

Gary started contributing to the project in June 2017. He helped

with

the

Flip-6 implementation, implemented many of the new REST handlers,

fixed

Mesos issues and initiated the Jepsen-based distributed test suite

which

uncovered several serious issues. Moreover, he actively helps

community

members on the mailing list and with PR reviews.

Please join me in congratulating Gary for becoming a Flink

committer!

Cheers,
Till

[jira] [Created] (FLINK-10302) Mark legacy(non-flip 6) code as Deprecated

[JIRA]

陈梓立 created FLINK-10302:
---

 Summary: Mark legacy(non-flip 6) code as Deprecated
 Key: FLINK-10302
 URL: https://issues.apache.org/jira/browse/FLINK-10302
 Project: Flink
  Issue Type: Improvement
Reporter: 陈梓立


There are several time I dash into some classes/methods, finding them weird 
from the FLIP-6 codebase and finally figure out that they are legacy codes.

Currently we mix up legacy code with FLIP-6 code in same place(i.e. some 
package), new contributor might casually lost into such code and result in 
works in vain.

With [FLINK-4319] closed we announced that FLIP-6 is production ready, and 
[~trohrm...@apache.org] comments on this 
[commits|https://github.com/apache/flink/commit/ddd6a99a95b56c52ea5b5153b7270b578f5479bc#commitcomment-30330739]
 shows that it is planned to remove legacy code.

I'd prefer to marking all legacy class as Deprecated for now thus our 
contributors could recognize them quickly and do not ruin they work in vain.

What do you think? cc [~Zentol]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New committer Gary Yao

[Ufuk Celebi]

Great addition to the committers. Congrats, Gary!

– Ufuk


On Fri, Sep 7, 2018 at 4:45 PM, Kostas Kloudas
 wrote:
> Congratulations Gary! Well deserved!
>
> Cheers,
> Kostas
>
>> On Sep 7, 2018, at 4:43 PM, Fabian Hueske  wrote:
>>
>> Congratulations Gary!
>>
>> 2018-09-07 16:29 GMT+02:00 Thomas Weise :
>>
>>> Congrats, Gary!
>>>
>>> On Fri, Sep 7, 2018 at 4:17 PM Dawid Wysakowicz 
>>> wrote:
>>>
 Congratulations Gary! Well deserved!

 On 07/09/18 16:00, zhangmingleihe wrote:
> Congrats Gary!
>
> Cheers
> Minglei
>
>> 在 2018年9月7日，下午9:59，Andrey Zagrebin  写道：
>>
>> Congratulations Gary!
>>
>>> On 7 Sep 2018, at 15:45, Stefan Richter >>>
 wrote:
>>>
>>> Congrats Gary!
>>>
 Am 07.09.2018 um 15:14 schrieb Till Rohrmann >>> :

 Hi everybody,

 On behalf of the PMC I am delighted to announce Gary Yao as a new
 Flink
 committer!

 Gary started contributing to the project in June 2017. He helped
>>> with
 the
 Flip-6 implementation, implemented many of the new REST handlers,
 fixed
 Mesos issues and initiated the Jepsen-based distributed test suite
 which
 uncovered several serious issues. Moreover, he actively helps
 community
 members on the mailing list and with PR reviews.

 Please join me in congratulating Gary for becoming a Flink
>>> committer!

 Cheers,
 Till
>


>>>
>

Re: [ANNOUNCE] New committer Gary Yao

[Kostas Kloudas]

Congratulations Gary! Well deserved!

Cheers,
Kostas

> On Sep 7, 2018, at 4:43 PM, Fabian Hueske  wrote:
> 
> Congratulations Gary!
> 
> 2018-09-07 16:29 GMT+02:00 Thomas Weise :
> 
>> Congrats, Gary!
>> 
>> On Fri, Sep 7, 2018 at 4:17 PM Dawid Wysakowicz 
>> wrote:
>> 
>>> Congratulations Gary! Well deserved!
>>> 
>>> On 07/09/18 16:00, zhangmingleihe wrote:
 Congrats Gary!
 
 Cheers
 Minglei
 
> 在 2018年9月7日，下午9:59，Andrey Zagrebin  写道：
> 
> Congratulations Gary!
> 
>> On 7 Sep 2018, at 15:45, Stefan Richter >> 
>>> wrote:
>> 
>> Congrats Gary!
>> 
>>> Am 07.09.2018 um 15:14 schrieb Till Rohrmann >> :
>>> 
>>> Hi everybody,
>>> 
>>> On behalf of the PMC I am delighted to announce Gary Yao as a new
>>> Flink
>>> committer!
>>> 
>>> Gary started contributing to the project in June 2017. He helped
>> with
>>> the
>>> Flip-6 implementation, implemented many of the new REST handlers,
>>> fixed
>>> Mesos issues and initiated the Jepsen-based distributed test suite
>>> which
>>> uncovered several serious issues. Moreover, he actively helps
>>> community
>>> members on the mailing list and with PR reviews.
>>> 
>>> Please join me in congratulating Gary for becoming a Flink
>> committer!
>>> 
>>> Cheers,
>>> Till
 
>>> 
>>> 
>> 

Re: [ANNOUNCE] New committer Gary Yao

[Fabian Hueske]

Congratulations Gary!

2018-09-07 16:29 GMT+02:00 Thomas Weise :

> Congrats, Gary!
>
> On Fri, Sep 7, 2018 at 4:17 PM Dawid Wysakowicz 
> wrote:
>
> > Congratulations Gary! Well deserved!
> >
> > On 07/09/18 16:00, zhangmingleihe wrote:
> > > Congrats Gary!
> > >
> > > Cheers
> > > Minglei
> > >
> > >> 在 2018年9月7日，下午9:59，Andrey Zagrebin  写道：
> > >>
> > >> Congratulations Gary!
> > >>
> > >>> On 7 Sep 2018, at 15:45, Stefan Richter  >
> > wrote:
> > >>>
> > >>> Congrats Gary!
> > >>>
> >  Am 07.09.2018 um 15:14 schrieb Till Rohrmann  >:
> > 
> >  Hi everybody,
> > 
> >  On behalf of the PMC I am delighted to announce Gary Yao as a new
> > Flink
> >  committer!
> > 
> >  Gary started contributing to the project in June 2017. He helped
> with
> > the
> >  Flip-6 implementation, implemented many of the new REST handlers,
> > fixed
> >  Mesos issues and initiated the Jepsen-based distributed test suite
> > which
> >  uncovered several serious issues. Moreover, he actively helps
> > community
> >  members on the mailing list and with PR reviews.
> > 
> >  Please join me in congratulating Gary for becoming a Flink
> committer!
> > 
> >  Cheers,
> >  Till
> > >
> >
> >
>

Re: [ANNOUNCE] New committer Gary Yao

[Thomas Weise]

Congrats, Gary!

On Fri, Sep 7, 2018 at 4:17 PM Dawid Wysakowicz 
wrote:

> Congratulations Gary! Well deserved!
>
> On 07/09/18 16:00, zhangmingleihe wrote:
> > Congrats Gary!
> >
> > Cheers
> > Minglei
> >
> >> 在 2018年9月7日，下午9:59，Andrey Zagrebin  写道：
> >>
> >> Congratulations Gary!
> >>
> >>> On 7 Sep 2018, at 15:45, Stefan Richter 
> wrote:
> >>>
> >>> Congrats Gary!
> >>>
>  Am 07.09.2018 um 15:14 schrieb Till Rohrmann :
> 
>  Hi everybody,
> 
>  On behalf of the PMC I am delighted to announce Gary Yao as a new
> Flink
>  committer!
> 
>  Gary started contributing to the project in June 2017. He helped with
> the
>  Flip-6 implementation, implemented many of the new REST handlers,
> fixed
>  Mesos issues and initiated the Jepsen-based distributed test suite
> which
>  uncovered several serious issues. Moreover, he actively helps
> community
>  members on the mailing list and with PR reviews.
> 
>  Please join me in congratulating Gary for becoming a Flink committer!
> 
>  Cheers,
>  Till
> >
>
>

Re: [ANNOUNCE] New committer Gary Yao

[Dawid Wysakowicz]

Congratulations Gary! Well deserved!

On 07/09/18 16:00, zhangmingleihe wrote:
> Congrats Gary!
>
> Cheers
> Minglei
>
>> 在 2018年9月7日，下午9:59，Andrey Zagrebin  写道：
>>
>> Congratulations Gary!
>>
>>> On 7 Sep 2018, at 15:45, Stefan Richter  wrote:
>>>
>>> Congrats Gary!
>>>
 Am 07.09.2018 um 15:14 schrieb Till Rohrmann :

 Hi everybody,

 On behalf of the PMC I am delighted to announce Gary Yao as a new Flink
 committer!

 Gary started contributing to the project in June 2017. He helped with the
 Flip-6 implementation, implemented many of the new REST handlers, fixed
 Mesos issues and initiated the Jepsen-based distributed test suite which
 uncovered several serious issues. Moreover, he actively helps community
 members on the mailing list and with PR reviews.

 Please join me in congratulating Gary for becoming a Flink committer!

 Cheers,
 Till
>

Re: [DISCUSS] Proposal of external shuffle service

[Till Rohrmann]

The rough plan sounds good Zhijiang. I think we should continue with what
you've proposed: Open a JIRA issue and creating a design document which
outlines the required changes a little bit more in detail. Once this is
done, we should link the design document in the JIRA issue and post it here
for further discussion.

Cheers,
Till

On Wed, Aug 29, 2018 at 6:04 PM Zhijiang(wangzhijiang999) <
wangzhijiang...@aliyun.com> wrote:

> Glad to receive your positive feedbacks Till!
>
> Actually our motivation is to support batch job well as you mentioned.
>
> For output level, flink already has the Subpartition abstraction(writer),
> and currently there are PipelinedSubpartition(memory output) and
> SpillableSubpartition(one-sp-one-file output) implementations. We can
> extend this abstraction to realize other persistent outputs (e.g.
> sort-merge-file).
>
> For transport level(shuffle service), the current SubpartitionView
> abstraction(reader) seems as the brige linked with the output level, then
> the view can understand and read the different output formats. The current
> NetworkEnvironment seems take the role of internal shuffle service in
> TaskManager and the transport server is realized by netty inside. This
> component can also be started in other external containers like NodeManager
> of yarn to take the role of external shuffle service. Further we can
> abstract to extend the shuffle service for transporting outputs by http or
> rdma instead of current netty.  This abstraction should provide the way for
> output registration in order to read the results correctly, similar with
> current SubpartitionView.
>
> The above is still a rough idea. Next I plan to create a feature jira to
> cover the related changes if possible. It would be better if getting help
> from related committers to review the detail designs together.
>
> Best,
> Zhijiang
>
> --
> 发件人：Till Rohrmann 
> 发送时间：2018年8月29日(星期三) 17:36
> 收件人：dev ; Zhijiang(wangzhijiang999) <
> wangzhijiang...@aliyun.com>
> 主 题：Re: [DISCUSS] Proposal of external shuffle service
>
> Thanks for starting this design discussion Zhijiang!
>
> I really like the idea to introduce a ShuffleService abstraction which
> allows to have different implementations depending on the actual use case.
> Especially for batch jobs I can clearly see the benefits of persisting the
> results somewhere else.
>
> Do you already know which interfaces we need to extend and where to
> introduce new abstractions?
>
> Cheers,
> Till
>
> On Mon, Aug 27, 2018 at 1:57 PM Zhijiang(wangzhijiang999)
>  wrote:
> Hi all!
>
> The shuffle service is responsible for transporting upstream produced data
> to the downstream side. In flink, the NettyServer is used for network
> transport service and this component is started in the TaskManager process.
> That means the TaskManager can support internal shuffle service which
> exists some concerns:
> 1. If a task finishes, the ResultPartition of this task still retains
> registered in TaskManager, because the output buffers have to be
> transported by internal shuffle service in TaskManager. That means the
> TaskManager can not be released by ResourceManager until ResultPartition
> released. It may waste container resources and can not support well for
> dynamic resource scenarios.
> 2. If we want to expand another shuffle service implementation, the
> current mechanism is not easy to handle, because the output level (result
> partition) and transport level (shuffle service) are not divided clearly
> and loss of abstraction to be extended.
>
> For above considerations, we propose the external shuffle service which
> can be deployed on any other external contaienrs, e.g. NodeManager
> container in yarn. Then the TaskManager can be released ASAP ifneeded when
> all the internal tasks finished. The persistent output files of these
> finished tasks can be served to transport by external shuffle service in
> the same machine.
>
> Further we can abstract both of the output level and transport level to
> support different implementations. e.g. We realized merging the data of all
> the subpartitions into limited persistent local files for disk improvements
> in some scenarios instead of one-subpartiton-one-file.
>
> I know it may be a big work for doing this, and I just point out some
> ideas, and wish getting any feedbacks from you!
>
> Best,
> Zhijiang
>
>
>

Re: [ANNOUNCE] New committer Gary Yao

[zhangmingleihe]

Congrats Gary!

Cheers
Minglei

> 在 2018年9月7日，下午9:59，Andrey Zagrebin  写道：
> 
> Congratulations Gary!
> 
>> On 7 Sep 2018, at 15:45, Stefan Richter  wrote:
>> 
>> Congrats Gary!
>> 
>>> Am 07.09.2018 um 15:14 schrieb Till Rohrmann :
>>> 
>>> Hi everybody,
>>> 
>>> On behalf of the PMC I am delighted to announce Gary Yao as a new Flink
>>> committer!
>>> 
>>> Gary started contributing to the project in June 2017. He helped with the
>>> Flip-6 implementation, implemented many of the new REST handlers, fixed
>>> Mesos issues and initiated the Jepsen-based distributed test suite which
>>> uncovered several serious issues. Moreover, he actively helps community
>>> members on the mailing list and with PR reviews.
>>> 
>>> Please join me in congratulating Gary for becoming a Flink committer!
>>> 
>>> Cheers,
>>> Till
>> 

Re: [ANNOUNCE] New committer Gary Yao

[Andrey Zagrebin]

Congratulations Gary!

> On 7 Sep 2018, at 15:45, Stefan Richter  wrote:
> 
> Congrats Gary!
> 
>> Am 07.09.2018 um 15:14 schrieb Till Rohrmann :
>> 
>> Hi everybody,
>> 
>> On behalf of the PMC I am delighted to announce Gary Yao as a new Flink
>> committer!
>> 
>> Gary started contributing to the project in June 2017. He helped with the
>> Flip-6 implementation, implemented many of the new REST handlers, fixed
>> Mesos issues and initiated the Jepsen-based distributed test suite which
>> uncovered several serious issues. Moreover, he actively helps community
>> members on the mailing list and with PR reviews.
>> 
>> Please join me in congratulating Gary for becoming a Flink committer!
>> 
>> Cheers,
>> Till
> 

Re: [ANNOUNCE] New committer Gary Yao

[Stefan Richter]

Congrats Gary!

> Am 07.09.2018 um 15:14 schrieb Till Rohrmann :
> 
> Hi everybody,
> 
> On behalf of the PMC I am delighted to announce Gary Yao as a new Flink
> committer!
> 
> Gary started contributing to the project in June 2017. He helped with the
> Flip-6 implementation, implemented many of the new REST handlers, fixed
> Mesos issues and initiated the Jepsen-based distributed test suite which
> uncovered several serious issues. Moreover, he actively helps community
> members on the mailing list and with PR reviews.
> 
> Please join me in congratulating Gary for becoming a Flink committer!
> 
> Cheers,
> Till

Re: [ANNOUNCE] New committer Gary Yao

[陈梓立]

Congratulation Gary!

Best,
tison.


Till Rohrmann  于2018年9月7日周五 下午9:15写道：

> Hi everybody,
>
> On behalf of the PMC I am delighted to announce Gary Yao as a new Flink
> committer!
>
> Gary started contributing to the project in June 2017. He helped with the
> Flip-6 implementation, implemented many of the new REST handlers, fixed
> Mesos issues and initiated the Jepsen-based distributed test suite which
> uncovered several serious issues. Moreover, he actively helps community
> members on the mailing list and with PR reviews.
>
> Please join me in congratulating Gary for becoming a Flink committer!
>
> Cheers,
> Till
>

[ANNOUNCE] New committer Gary Yao

[Till Rohrmann]

Hi everybody,

On behalf of the PMC I am delighted to announce Gary Yao as a new Flink
committer!

Gary started contributing to the project in June 2017. He helped with the
Flip-6 implementation, implemented many of the new REST handlers, fixed
Mesos issues and initiated the Jepsen-based distributed test suite which
uncovered several serious issues. Moreover, he actively helps community
members on the mailing list and with PR reviews.

Please join me in congratulating Gary for becoming a Flink committer!

Cheers,
Till

[jira] [Created] (FLINK-10301) Allow a custom Configuration in StreamNetworkBenchmarkEnvironment

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-10301:
---

 Summary: Allow a custom Configuration in 
StreamNetworkBenchmarkEnvironment
 Key: FLINK-10301
 URL: https://issues.apache.org/jira/browse/FLINK-10301
 Project: Flink
  Issue Type: Improvement
  Components: Network, Tests
Affects Versions: 1.7.0
Reporter: Nico Kruber
Assignee: Nico Kruber


{{StreamNetworkBenchmarkEnvironment}} currently instantiates its 
{{NettyConfig}} with {{new Configuration()}} but for testing with different 
options, it would be nice to allow the user to provide a custom instance 
instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10300) Prometheus job level metrics not removed after job finished

[Helmut Zechmann (JIRA)]

Helmut Zechmann created FLINK-10300:
---

 Summary: Prometheus job level metrics not removed after job 
finished
 Key: FLINK-10300
 URL: https://issues.apache.org/jira/browse/FLINK-10300
 Project: Flink
  Issue Type: Bug
  Components: Metrics
Affects Versions: 1.5.3
Reporter: Helmut Zechmann


Flink provides job level metrics for flink jobs. After a job is finished those 
metrics should be removed, else we run into problems when many jobs are 
executed on a cluster.

How to reproduce this:

Setup:
 * flink 1.5.3 in standalone mode
 * 1 JobManager
 * 1 TaskManager
 * flink-metrics-prometheus-1.5.3.jar in lib dir

Metrics config:
{code:java}
metrics.reporters: prom 
metrics.reporter.prom.class: 
org.apache.flink.metrics.prometheus.PrometheusReporter 
metrics.reporter.prom.port: 7000-7001
{code}
 Run the wordcount job. After running the job, job related metrics are still 
available:

 
{code:java}
flink_taskmanager_Status_JVM_GarbageCollector_G1_Old_Generation_Count{tm_id="ee893c28f70d285e701f838706ce8810",host="helmuts_mbp",}
 1.0
# HELP flink_taskmanager_job_task_operator_numRecordsOutPerSecond 
numRecordsOutPerSecond (scope: taskmanager_job_task_operator)
# TYPE flink_taskmanager_job_task_operator_numRecordsOutPerSecond gauge
flink_taskmanager_job_task_operator_numRecordsOutPerSecond{job_id="2a7c77aacf6b18da389189a3bae6ff48",task_id="529e7a1eaba520b18dc7864f821ada08",task_attempt_id="3bc0d07eb56df676b088a8ec13531c98",host="helmuts_mbp",operator_id="529e7a1eaba520b18dc7864f821ada08",operator_name="DataSource__at_getDefaultTextLineDataSet_WordCountData_java:70___org_apache_flin",task_name="CHAIN_DataSource__at_getDefaultTextLineDataSet_WordCountData_java:70___org_apache_flink_api_java_io_CollectionInputFormat__FlatMap__FlatMap_at_main_WordCount_java:77__Combine__SUM_1___at_main_WordCount_java:80_",task_attempt_num="0",job_name="Flink_Java_Job_at_Fri_Sep_07_13:00:12_CEST_2018",tm_id="ee893c28f70d285e701f838706ce8810",subtask_index="0",}
 0.0
...
{code}
 
 With each finished job the prometheus output gets bigger and bigger until the 
prometheus output fails to load.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10299) RowSerializer.copy data value cast exception

[ambition (JIRA)]

ambition created FLINK-10299:


 Summary: RowSerializer.copy data value cast exception
 Key: FLINK-10299
 URL: https://issues.apache.org/jira/browse/FLINK-10299
 Project: Flink
  Issue Type: Bug
  Components: Core
Affects Versions: 1.6.0
Reporter: ambition
 Attachments: image-2018-09-07-17-47-04-343.png

Flink sql deal with User behavior data collection, such as：
{code:java}
{
    "event_id": "session_start",
    "timestamp": "-",    // error data,
    "viewport_height": "667",
     "viewport_width": "-"    //error data
}
{code}
Causing exception info :
{code:java}
2018-09-07 10:47:01,834 [flink-akka.actor.default-dispatcher-2] INFO 
executiongraph.ExecutionGraph (ExecutionGraph.java:tryRestartOrFail(1511)) - 
Could not restart the job Flink Streaming Job 
(6f0248219c631158f6e38f2dca0beb91) because the restart strategy prevented it.
java.lang.ClassCastException: java.lang.String cannot be cast to 
java.sql.Timestamp
at 
org.apache.flink.api.common.typeutils.base.SqlTimestampSerializer.copy(SqlTimestampSerializer.java:27)
at 
org.apache.flink.api.java.typeutils.runtime.RowSerializer.copy(RowSerializer.java:95)
at 
org.apache.flink.api.java.typeutils.runtime.RowSerializer.copy(RowSerializer.java:46)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:577)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:689)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:667)
at 
org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
at 
org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:111)
at 
org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:398)
at 
org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:89)
at 
org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:154)
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:738)
at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:56)
at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:99)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:748)
2018-09-07 10:47:01,834 [flink-akka.actor.default-dispatcher-2] INFO 
checkpoint.CheckpointCoordinator (CheckpointCoordinator.java:shutdown(320)) - 
Stopping checkpoint coordinator for job 6f0248219c631158f6e38f2dca0beb91.
2018-09-07 10:47:01,834 [flink-akka.actor.default-dispatcher-2] INFO 
checkpoint.StandaloneCompletedCheckpointStore 
(StandaloneCompletedCheckpointStore.java:shutdown(102)) - Shutting down
{code}
we use Flink  checkpoint function and Uncatch exception lead to  Could not 
restart this job,  so we just simple ,hope flink commiter provide better 
solution。

!image-2018-09-07-17-47-04-343.png!

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10298) Batch Job Failover Strategy

[JIN SUN (JIRA)]

JIN SUN created FLINK-10298:
---

 Summary: Batch Job Failover Strategy
 Key: FLINK-10298
 URL: https://issues.apache.org/jira/browse/FLINK-10298
 Project: Flink
  Issue Type: Sub-task
  Components: JobManager
Reporter: JIN SUN
Assignee: JIN SUN


The new failover strategy needs to consider handling failures according to 
different failure types. It orchestrates all the logics we mentioned in this 
[document|https://docs.google.com/document/d/1FdZdcA63tPUEewcCimTFy9Iz2jlVlMRANZkO4RngIuk/edit#],
 we can put the logic in onTaskFailure method of the FailoverStrategy 
interface, with the logic inline:
{code:java}
public void onTaskFailure(Execution taskExecution, Throwable cause) {  

    //1. Get the throwable type

    //2. If the type is NonrecoverableType fail the job

    //3. If the type is PatritionDataMissingError, do revocation

    //4. If the type is EnvironmentError, do check blacklist

//5. Other failure types are recoverable, but we need to remember the count of 
the failure,

if it exceeds the threshold, fail the job

}{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10297) PostVersionedIOReadableWritable ignores result of InputStream.read(...)

[Stefan Richter (JIRA)]

Stefan Richter created FLINK-10297:
--

 Summary: PostVersionedIOReadableWritable ignores result of 
InputStream.read(...)
 Key: FLINK-10297
 URL: https://issues.apache.org/jira/browse/FLINK-10297
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing
Affects Versions: 1.6.0, 1.5.3, 1.4.2
Reporter: Stefan Richter


PostVersionedIOReadableWritable ignores result of {{InputStream.read(...)}}. 
Probably the intention was to invoke {{readFully}}. As it is now, this can lead 
to a corrupted deserialization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)