[GitHub] flink pull request #5792: [Flink-8563][Table API & SQL] add unittest for con...

[suez1224]

Github user suez1224 closed the pull request at:

https://github.com/apache/flink/pull/5792


---

[GitHub] flink pull request #5792: [Flink-8563][Table API & SQL] add unittest for con...

[suez1224]

GitHub user suez1224 reopened a pull request:

https://github.com/apache/flink/pull/5792

[Flink-8563][Table API & SQL] add unittest for consecutive dot access of 
composite array element in SQL

 ## What is the purpose of the change

add unittest for consecutive dot access of composite array element in 
SQL. This depends on https://github.com/apache/flink/pull/5791.


## Brief change log

  -  add unittest


## Verifying this change

This change is already covered by existing tests.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (y no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/suez1224/flink FLINK-8563

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5792.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5792


commit 637b7462ac049d347fb5bbb8e68ed4fca7b0ec93
Author: Shuyi Chen 
Date:   2018-03-29T18:39:09Z

upgrade calcite dependency to 1.16

commit aea021cb9efc869872595f64467f8d2ec8071ea4
Author: Shuyi Chen 
Date:   2018-03-29T19:15:15Z

add unittest for consecutive dot access of composite array element in SQL




---

[GitHub] flink issue #5792: [Flink-8563][Table API & SQL] add unittest for consecutiv...

[suez1224]

Github user suez1224 commented on the issue:

https://github.com/apache/flink/pull/5792
  
retry build


---

[jira] [Updated] (FLINK-9105) Table program compiles failed

[Bob Lau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bob Lau updated FLINK-9105:
---
Component/s: Table API & SQL

> Table program compiles failed
> -
>
> Key: FLINK-9105
> URL: https://issues.apache.org/jira/browse/FLINK-9105
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API, Table API  SQL
>Affects Versions: 1.5.0
>Reporter: Bob Lau
>Priority: Major
>
> ExceptionStack:
> org.apache.flink.client.program.ProgramInvocationException: 
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be 
> compiled. This is a bug. Please file an issue.
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:253)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:463)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:456)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.executeRemotely(RemoteStreamEnvironment.java:219)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.execute(RemoteStreamEnvironment.java:178)
> at 
> com.tydic.tysc.job.service.SubmitJobService.submitJobToStandaloneCluster(SubmitJobService.java:150)
> at 
> com.tydic.tysc.rest.SubmitJobController.submitJob(SubmitJobController.java:55)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
> at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> com.tydic.tysc.filter.ShiroSessionFilter.doFilter(ShiroSessionFilter.java:51)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
> at com.tydic.tysc.filter.OauthFilter.doFilter(OauthFilter.java:48)
> at 
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
> at 
> 

[jira] [Updated] (FLINK-9105) Table program compiles failed

[Bob Lau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bob Lau updated FLINK-9105:
---
Affects Version/s: (was: 1.4.2)
   (was: 1.4.1)
   (was: 1.4.0)

> Table program compiles failed
> -
>
> Key: FLINK-9105
> URL: https://issues.apache.org/jira/browse/FLINK-9105
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API, Table API  SQL
>Affects Versions: 1.5.0
>Reporter: Bob Lau
>Priority: Major
>
> ExceptionStack:
> org.apache.flink.client.program.ProgramInvocationException: 
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be 
> compiled. This is a bug. Please file an issue.
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:253)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:463)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:456)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.executeRemotely(RemoteStreamEnvironment.java:219)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.execute(RemoteStreamEnvironment.java:178)
> at 
> com.tydic.tysc.job.service.SubmitJobService.submitJobToStandaloneCluster(SubmitJobService.java:150)
> at 
> com.tydic.tysc.rest.SubmitJobController.submitJob(SubmitJobController.java:55)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
> at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> com.tydic.tysc.filter.ShiroSessionFilter.doFilter(ShiroSessionFilter.java:51)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
> at com.tydic.tysc.filter.OauthFilter.doFilter(OauthFilter.java:48)
> at 
> 

[jira] [Comment Edited] (FLINK-9105) Table program compiles failed

[Bob Lau (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420044#comment-16420044
 ] 

Bob Lau edited comment on FLINK-9105 at 3/30/18 2:52 AM:
-

[~twalthr]Thank you for your response!

The SQL statement I'm going to compile is  '''  insert into zdry_wbyj select 
a.certificate_code,a.user_name from wb_swry a inner join  ry_bc_duibi_all b on 
a.certificate_code = b.zjhm where b.bc_end_time > a.offline_time and b.yxx=1  
''' ..

I'm using a stream environment API，as follows：

StreamExecutionEnvironment environment = 
StreamExecutionEnvironment.createRemoteEnvironment(host, port, depend);

The variable 'depend'  is an array of jar package absolute paths that contain 
all of the current project.

My current operating system is Mac OS X, the standalone flink cluster is CentOS 
6.8.

The jar packages of the current project is compiled in mac os x,  the remote 
flink cluster is built in centos 6.8

Do I need to copy the Linux's flink jar package to the current project, and 
depend them on maven of current project?

It's surprising that I can debug success in the local mode.

 

 


was (Author: bob365):
[~twalthr]Thank you for your response!

The SQL statement I'm going to compile is'''insert into zdry_wbyj select 
a.certificate_code,a.user_name from wb_swry a inner join  ry_bc_duibi_all b on 
a.certificate_code = b.zjhm where b.bc_end_time > a.offline_time and 
b.yxx=1'''..

I'm using a stream environment API，as follows：

StreamExecutionEnvironment environment = 
StreamExecutionEnvironment.createRemoteEnvironment(host, port, depend);

The variable 'depend'  is an array of jar package absolute paths that contain 
all of the current project.

My current operating system is Mac OS X, the standalone flink cluster is CentOS 
6.8.

The jar packages of the current project is compiled in mac os x,  the remote 
flink cluster is built in centos 6.8

Do I need to copy the Linux's flink jar package to the current project, and 
depend them on maven of current project?

It's surprising that I can debug success in the local mode.

 

 

> Table program compiles failed
> -
>
> Key: FLINK-9105
> URL: https://issues.apache.org/jira/browse/FLINK-9105
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.4.0, 1.5.0, 1.4.1, 1.4.2
>Reporter: Bob Lau
>Priority: Major
>
> ExceptionStack:
> org.apache.flink.client.program.ProgramInvocationException: 
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be 
> compiled. This is a bug. Please file an issue.
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:253)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:463)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:456)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.executeRemotely(RemoteStreamEnvironment.java:219)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.execute(RemoteStreamEnvironment.java:178)
> at 
> com.tydic.tysc.job.service.SubmitJobService.submitJobToStandaloneCluster(SubmitJobService.java:150)
> at 
> com.tydic.tysc.rest.SubmitJobController.submitJob(SubmitJobController.java:55)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at 

[jira] [Commented] (FLINK-8205) Multi key get

[Sihua Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420071#comment-16420071
 ] 

Sihua Zhou commented on FLINK-8205:
---

Hi guys, this issue seems to be inactive for some time, I'd like to take this 
ticket if no one object, and I think we can't get it into 1.5 now, since 1.5 is 
being releasing...I want to fixed it in 1.5.x or 1.6, what do you think 
[~kkl0u] ?

> Multi key get
> -
>
> Key: FLINK-8205
> URL: https://issues.apache.org/jira/browse/FLINK-8205
> Project: Flink
>  Issue Type: New Feature
>  Components: Queryable State
>Affects Versions: 1.4.0
> Environment: Any
>Reporter: Martin Eden
>Priority: Major
> Fix For: 1.5.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Currently the Java queryable state api only allows for fetching one key at a 
> time. It would be extremely useful and more efficient if a similar call 
> exists for submitting multiple keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (FLINK-9105) Table program compiles failed

[Bob Lau (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420044#comment-16420044
 ] 

Bob Lau edited comment on FLINK-9105 at 3/30/18 1:55 AM:
-

[~twalthr]Thank you for your response!

The SQL statement I'm going to compile is'''insert into zdry_wbyj select 
a.certificate_code,a.user_name from wb_swry a inner join  ry_bc_duibi_all b on 
a.certificate_code = b.zjhm where b.bc_end_time > a.offline_time and 
b.yxx=1'''..

I'm using a stream environment API，as follows：

StreamExecutionEnvironment environment = 
StreamExecutionEnvironment.createRemoteEnvironment(host, port, depend);

The variable 'depend'  is an array of jar package absolute paths that contain 
all of the current project.

My current operating system is Mac OS X, the standalone flink cluster is CentOS 
6.8.

The jar packages of the current project is compiled in mac os x,  the remote 
flink cluster is built in centos 6.8

Do I need to copy the Linux's flink jar package to the current project, and 
depend them on maven of current project?

It's surprising that I can debug success in the local mode.

 

 


was (Author: bob365):
@[~twalthr]Thank you for your response!

The SQL statement I'm going to compile is'''insert into zdry_wbyj select 
a.certificate_code,a.user_name from wb_swry a inner join  ry_bc_duibi_all b on 
a.certificate_code = b.zjhm where b.bc_end_time > a.offline_time and 
b.yxx=1'''..

I'm using a stream environment API，as follows：

StreamExecutionEnvironment environment = 
StreamExecutionEnvironment.createRemoteEnvironment(host, port, depend);

The variable 'depend'  is an array of jar package absolute paths that contain 
all of the current project.

My current operating system is Mac OS X, the standalone flink cluster is CentOS 
6.8.

The jar packages of the current project is compiled in mac os x,  the remote 
flink cluster is built in centos 6.8

Do I need to copy the Linux's flink jar package to the current project, and 
depend them on maven of current project?

 

 

> Table program compiles failed
> -
>
> Key: FLINK-9105
> URL: https://issues.apache.org/jira/browse/FLINK-9105
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.4.0, 1.5.0, 1.4.1, 1.4.2
>Reporter: Bob Lau
>Priority: Major
>
> ExceptionStack:
> org.apache.flink.client.program.ProgramInvocationException: 
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be 
> compiled. This is a bug. Please file an issue.
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:253)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:463)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:456)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.executeRemotely(RemoteStreamEnvironment.java:219)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.execute(RemoteStreamEnvironment.java:178)
> at 
> com.tydic.tysc.job.service.SubmitJobService.submitJobToStandaloneCluster(SubmitJobService.java:150)
> at 
> com.tydic.tysc.rest.SubmitJobController.submitJob(SubmitJobController.java:55)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
> at 
> 

[jira] [Assigned] (FLINK-8825) Disallow new String() without charset in checkstyle

[vinoyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

vinoyang reassigned FLINK-8825:
---

Assignee: vinoyang

> Disallow new String() without charset in checkstyle
> ---
>
> Key: FLINK-8825
> URL: https://issues.apache.org/jira/browse/FLINK-8825
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: vinoyang
>Priority: Major
> Fix For: 1.5.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-7775) Remove unreferenced method PermanentBlobCache#getNumberOfCachedJobs

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-7775:
--
Description: 
{code}
  public int getNumberOfCachedJobs() {
return jobRefCounters.size();
  }
{code}

The method of PermanentBlobCache is not used.
We should remove it.

  was:
{code}
  public int getNumberOfCachedJobs() {
return jobRefCounters.size();
  }
{code}
The method of PermanentBlobCache is not used.
We should remove it.


> Remove unreferenced method PermanentBlobCache#getNumberOfCachedJobs
> ---
>
> Key: FLINK-7775
> URL: https://issues.apache.org/jira/browse/FLINK-7775
> Project: Flink
>  Issue Type: Task
>  Components: Local Runtime
>Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Minor
>
> {code}
>   public int getNumberOfCachedJobs() {
> return jobRefCounters.size();
>   }
> {code}
> The method of PermanentBlobCache is not used.
> We should remove it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (FLINK-9105) Table program compiles failed

[Bob Lau (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420044#comment-16420044
 ] 

Bob Lau edited comment on FLINK-9105 at 3/30/18 1:26 AM:
-

@[~twalthr]Thank you for your response!

The SQL statement I'm going to compile is'''insert into zdry_wbyj select 
a.certificate_code,a.user_name from wb_swry a inner join  ry_bc_duibi_all b on 
a.certificate_code = b.zjhm where b.bc_end_time > a.offline_time and 
b.yxx=1'''..

I'm using a stream environment API，as follows：

StreamExecutionEnvironment environment = 
StreamExecutionEnvironment.createRemoteEnvironment(host, port, depend);

The variable 'depend'  is an array of jar package absolute paths that contain 
all of the current project.

My current operating system is Mac OS X, the standalone flink cluster is CentOS 
6.8.

The jar packages of the current project is compiled in mac os x,  the remote 
flink cluster is built in centos 6.8

Do I need to copy the Linux's flink jar package to the current project, and 
depend them on maven of current project?

 

 


was (Author: bob365):
Thank you for your response!

The SQL statement I'm going to compile is'''insert into zdry_wbyj select 
a.certificate_code,a.user_name from wb_swry a inner join  ry_bc_duibi_all b on 
a.certificate_code = b.zjhm where b.bc_end_time > a.offline_time and 
b.yxx=1'''..

I'm using a stream environment API，as follows：

StreamExecutionEnvironment environment = 
StreamExecutionEnvironment.createRemoteEnvironment(host, port, depend);

The variable 'depend'  is an array of jar package absolute paths that contain 
all of the current project.

My current operating system is Mac OS X, the standalone flink cluster is CentOS 
6.8.

The jar packages of the current project is compiled in mac os x,  the remote 
flink cluster is built in centos 6.8

Do I need to copy the Linux's flink jar package to the current project, and 
depend them on maven of current project?

 

 

> Table program compiles failed
> -
>
> Key: FLINK-9105
> URL: https://issues.apache.org/jira/browse/FLINK-9105
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.4.0, 1.5.0, 1.4.1, 1.4.2
>Reporter: Bob Lau
>Priority: Major
>
> ExceptionStack:
> org.apache.flink.client.program.ProgramInvocationException: 
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be 
> compiled. This is a bug. Please file an issue.
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:253)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:463)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:456)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.executeRemotely(RemoteStreamEnvironment.java:219)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.execute(RemoteStreamEnvironment.java:178)
> at 
> com.tydic.tysc.job.service.SubmitJobService.submitJobToStandaloneCluster(SubmitJobService.java:150)
> at 
> com.tydic.tysc.rest.SubmitJobController.submitJob(SubmitJobController.java:55)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
> at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at 

[jira] [Commented] (FLINK-9105) Table program compiles failed

[Bob Lau (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420044#comment-16420044
 ] 

Bob Lau commented on FLINK-9105:


Thank you for your response!

The SQL statement I'm going to compile is'''insert into zdry_wbyj select 
a.certificate_code,a.user_name from wb_swry a inner join  ry_bc_duibi_all b on 
a.certificate_code = b.zjhm where b.bc_end_time > a.offline_time and 
b.yxx=1'''..

I'm using a stream environment API，as follows：

StreamExecutionEnvironment environment = 
StreamExecutionEnvironment.createRemoteEnvironment(host, port, depend);

The variable 'depend'  is an array of jar package absolute paths that contain 
all of the current project.

My current operating system is Mac OS X, the standalone flink cluster is CentOS 
6.8.

The jar packages of the current project is compiled in mac os x,  the remote 
flink cluster is built in centos 6.8

Do I need to copy the Linux's flink jar package to the current project, and 
depend them on maven of current project?

 

 

> Table program compiles failed
> -
>
> Key: FLINK-9105
> URL: https://issues.apache.org/jira/browse/FLINK-9105
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.4.0, 1.5.0, 1.4.1, 1.4.2
>Reporter: Bob Lau
>Priority: Major
>
> ExceptionStack:
> org.apache.flink.client.program.ProgramInvocationException: 
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be 
> compiled. This is a bug. Please file an issue.
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:253)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:463)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:456)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.executeRemotely(RemoteStreamEnvironment.java:219)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.execute(RemoteStreamEnvironment.java:178)
> at 
> com.tydic.tysc.job.service.SubmitJobService.submitJobToStandaloneCluster(SubmitJobService.java:150)
> at 
> com.tydic.tysc.rest.SubmitJobController.submitJob(SubmitJobController.java:55)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
> at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> 

[jira] [Created] (FLINK-9115) Support addition of part suffix in BucketingSink

[Lakshmi Rao (JIRA)]

Lakshmi Rao created FLINK-9115:
--

 Summary: Support addition of part suffix in BucketingSink
 Key: FLINK-9115
 URL: https://issues.apache.org/jira/browse/FLINK-9115
 Project: Flink
  Issue Type: Improvement
  Components: filesystem-connector
Reporter: Lakshmi Rao


Currently the BucketingSink allows addition of part prefix, pending 
prefix/suffix and in-progress prefix/suffix via setter methods. Can we also 
support setting part suffixes?


An instance where this maybe useful: I am currently writing GZIP compressed 
output to S3 using the BucketingSink and I would want the uploaded files to 
have a ".gz" or ".zip" extensions . An easy way to do this would be by setting  
a part file suffix with the required file extension. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9114) Enable user-provided, custom CheckpointRecoveryFactory for non-HA deployments

[Jacob Park (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419992#comment-16419992
 ] 

Jacob Park commented on FLINK-9114:
---

[~ecanzonieri] FYI.

> Enable user-provided, custom CheckpointRecoveryFactory for non-HA deployments
> -
>
> Key: FLINK-9114
> URL: https://issues.apache.org/jira/browse/FLINK-9114
> Project: Flink
>  Issue Type: Improvement
>  Components: Configuration, State Backends, Checkpointing
>Reporter: Jacob Park
>Assignee: Jacob Park
>Priority: Major
>
> When you operate a Flink application that uses externalized checkpoints to 
> S3, it becomes difficult to determine which checkpoint is the latest to 
> recover from. Because S3 provides read-after-write consistency only for PUTS, 
> listing a S3 path is not guaranteed to be consistent, so we do not know what 
> checkpoint to recover from.
> The goal of this improvement is to allow users to provide a custom 
> CheckpointRecoveryFactory for non-HA deployments such that we can use this 
> feature to fail checkpoints if we cannot guarantee we will know where a 
> checkpoint will be in S3, and co-publish checkpoint metadata to a strongly 
> consistent data store.
> I propose the following changes:
>  # Modify AbstractNonHaServices and StandaloneHaServices to accept an 
> Executor for the custom CheckpointRecoveryFactory.
>  # Create a CheckpointRecoveryFactoryLoader to provide the custom 
> CheckpointRecoveryFactory from configurations.
>  # Add new configurations for this feature.
> We considered the pluggable StateBackend and potential pluggable 
> HighAvailabilityServices. These were too convoluted to solve our problem, so 
> we would like custom CheckpointRecoveryFactory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9114) Enable user-provided, custom CheckpointRecoveryFactory for non-HA deployments

[Jacob Park (JIRA)]

Jacob Park created FLINK-9114:
-

 Summary: Enable user-provided, custom CheckpointRecoveryFactory 
for non-HA deployments
 Key: FLINK-9114
 URL: https://issues.apache.org/jira/browse/FLINK-9114
 Project: Flink
  Issue Type: Improvement
  Components: Configuration, State Backends, Checkpointing
Reporter: Jacob Park
Assignee: Jacob Park


When you operate a Flink application that uses externalized checkpoints to S3, 
it becomes difficult to determine which checkpoint is the latest to recover 
from. Because S3 provides read-after-write consistency only for PUTS, listing a 
S3 path is not guaranteed to be consistent, so we do not know what checkpoint 
to recover from.

The goal of this improvement is to allow users to provide a custom 
CheckpointRecoveryFactory for non-HA deployments such that we can use this 
feature to fail checkpoints if we cannot guarantee we will know where a 
checkpoint will be in S3, and co-publish checkpoint metadata to a strongly 
consistent data store.

I propose the following changes:
 # Modify AbstractNonHaServices and StandaloneHaServices to accept an Executor 
for the custom CheckpointRecoveryFactory.
 # Create a CheckpointRecoveryFactoryLoader to provide the custom 
CheckpointRecoveryFactory from configurations.
 # Add new configurations for this feature.

We considered the pluggable StateBackend and potential pluggable 
HighAvailabilityServices. These were too convoluted to solve our problem, so we 
would like custom CheckpointRecoveryFactory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8533) Support MasterTriggerRestoreHook state reinitialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419889#comment-16419889
 ] 

ASF GitHub Bot commented on FLINK-8533:
---

Github user EronWright commented on the issue:

https://github.com/apache/flink/pull/5427
  
@tillrohrmann @StephanEwen sorry about the long delay here, would you 
please take another look?

I followed Stephan's suggestion of not introducing a new method.   However, 
the semantics that I was shooting for with `initializeState` is that it would 
be called on both _start_ and _restart_.  I adjusted `JobManager` to call 
`restoreLatestCheckpointedState` on first execution (as does `JobMaster`).  Are 
you OK with that?


> Support MasterTriggerRestoreHook state reinitialization
> ---
>
> Key: FLINK-8533
> URL: https://issues.apache.org/jira/browse/FLINK-8533
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Eron Wright 
>Assignee: Eron Wright 
>Priority: Major
>
> {{MasterTriggerRestoreHook}} enables coordination with an external system for 
> taking or restoring checkpoints. When execution is restarted from a 
> checkpoint, {{restoreCheckpoint}} is called to restore or reinitialize the 
> external system state. There's an edge case where the external state is not 
> adequately reinitialized, that is when execution fails _before the first 
> checkpoint_. In that case, the hook is not invoked and has no opportunity to 
> restore the external state to initial conditions.
> The impact is a loss of exactly-once semantics in this case. For example, in 
> the Pravega source function, the reader group state (e.g. stream position 
> data) is stored externally. In the normal restore case, the reader group 
> state is forcibly rewound to the checkpointed position. In the edge case 
> where no checkpoint has yet been successful, the reader group state is not 
> rewound and consequently some amount of stream data is not reprocessed.
> A possible fix would be to introduce an {{initializeState}} method on the 
> hook interface. Similar to {{CheckpointedFunction::initializeState}}, this 
> method would be invoked unconditionally upon hook initialization. The Pravega 
> hook would, for example, initialize or forcibly reinitialize the reader group 
> state.    



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5427: [FLINK-8533] [checkpointing] Support MasterTriggerRestore...

[EronWright]

Github user EronWright commented on the issue:

https://github.com/apache/flink/pull/5427
  
@tillrohrmann @StephanEwen sorry about the long delay here, would you 
please take another look?

I followed Stephan's suggestion of not introducing a new method.   However, 
the semantics that I was shooting for with `initializeState` is that it would 
be called on both _start_ and _restart_.  I adjusted `JobManager` to call 
`restoreLatestCheckpointedState` on first execution (as does `JobMaster`).  Are 
you OK with that?


---

[jira] [Commented] (FLINK-8507) Upgrade Calcite dependency to 1.16

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419797#comment-16419797
 ] 

ASF GitHub Bot commented on FLINK-8507:
---

GitHub user suez1224 reopened a pull request:

https://github.com/apache/flink/pull/5791

[FLINK-8507][Table API & SQL] upgrade calcite dependency to 1.16


## What is the purpose of the change

Upgrade Flink table's Calcite dependency to 1.16. 


## Brief change log

  - Update pom.
  - Fix HepPlanner issue.


## Verifying this change

This change is already covered by existing tests, such as *(please describe 
tests)*.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (y no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/suez1224/flink FLINK-8507

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5791.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5791


commit 637b7462ac049d347fb5bbb8e68ed4fca7b0ec93
Author: Shuyi Chen 
Date:   2018-03-29T18:39:09Z

upgrade calcite dependency to 1.16




> Upgrade Calcite dependency to 1.16
> --
>
> Key: FLINK-8507
> URL: https://issues.apache.org/jira/browse/FLINK-8507
> Project: Flink
>  Issue Type: Task
>  Components: Table API  SQL
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8507) Upgrade Calcite dependency to 1.16

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419795#comment-16419795
 ] 

ASF GitHub Bot commented on FLINK-8507:
---

Github user suez1224 commented on the issue:

https://github.com/apache/flink/pull/5791
  
Close and reopen to trigger travis. 
SavepointITCase.testSavepointForJobWithIteration is unstable.


> Upgrade Calcite dependency to 1.16
> --
>
> Key: FLINK-8507
> URL: https://issues.apache.org/jira/browse/FLINK-8507
> Project: Flink
>  Issue Type: Task
>  Components: Table API  SQL
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5791: [FLINK-8507][Table API & SQL] upgrade calcite dependency ...

[suez1224]

Github user suez1224 commented on the issue:

https://github.com/apache/flink/pull/5791
  
Close and reopen to trigger travis. 
SavepointITCase.testSavepointForJobWithIteration is unstable.


---

[GitHub] flink pull request #5791: [FLINK-8507][Table API & SQL] upgrade calcite depe...

[suez1224]

GitHub user suez1224 reopened a pull request:

https://github.com/apache/flink/pull/5791

[FLINK-8507][Table API & SQL] upgrade calcite dependency to 1.16


## What is the purpose of the change

Upgrade Flink table's Calcite dependency to 1.16. 


## Brief change log

  - Update pom.
  - Fix HepPlanner issue.


## Verifying this change

This change is already covered by existing tests, such as *(please describe 
tests)*.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (y no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/suez1224/flink FLINK-8507

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5791.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5791


commit 637b7462ac049d347fb5bbb8e68ed4fca7b0ec93
Author: Shuyi Chen 
Date:   2018-03-29T18:39:09Z

upgrade calcite dependency to 1.16




---

[GitHub] flink pull request #5791: [FLINK-8507][Table API & SQL] upgrade calcite depe...

[suez1224]

Github user suez1224 closed the pull request at:

https://github.com/apache/flink/pull/5791


---

[jira] [Commented] (FLINK-8507) Upgrade Calcite dependency to 1.16

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419796#comment-16419796
 ] 

ASF GitHub Bot commented on FLINK-8507:
---

Github user suez1224 closed the pull request at:

https://github.com/apache/flink/pull/5791


> Upgrade Calcite dependency to 1.16
> --
>
> Key: FLINK-8507
> URL: https://issues.apache.org/jira/browse/FLINK-8507
> Project: Flink
>  Issue Type: Task
>  Components: Table API  SQL
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5794: [Flink-8509][Table API & SQL] Remove SqlGroupedWin...

[suez1224]

GitHub user suez1224 opened a pull request:

https://github.com/apache/flink/pull/5794

[Flink-8509][Table API & SQL] Remove SqlGroupedWindowFunction copy from 
flink

 ## What is the purpose of the change

Remove SqlGroupedWindowFunction copy from flink. This depends on 
https://github.com/apache/flink/pull/5791.


## Brief change log

  -  remove code.


## Verifying this change

This change is already covered by existing tests.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (y no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/suez1224/flink FLINK-8509

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5794.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5794


commit 637b7462ac049d347fb5bbb8e68ed4fca7b0ec93
Author: Shuyi Chen 
Date:   2018-03-29T18:39:09Z

upgrade calcite dependency to 1.16

commit 379111d0f2a8e2e54b5370884742fa5615d39b68
Author: Shuyi Chen 
Date:   2018-03-29T18:48:28Z

Remove SqlGroupedWindowFunction from Flink repo




---

[GitHub] flink pull request #5793: [Flink-8508][Table API & SQL] Remove RexSimplify c...

[suez1224]

GitHub user suez1224 opened a pull request:

https://github.com/apache/flink/pull/5793

[Flink-8508][Table API & SQL] Remove RexSimplify copy from flink

 ## What is the purpose of the change

Remove RexSimplify copy from flink. This depends on 
https://github.com/apache/flink/pull/5791.


## Brief change log

  -  remove code.


## Verifying this change

This change is already covered by existing tests.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (y no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/suez1224/flink FLINK-8508

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5793.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5793


commit 637b7462ac049d347fb5bbb8e68ed4fca7b0ec93
Author: Shuyi Chen 
Date:   2018-03-29T18:39:09Z

upgrade calcite dependency to 1.16

commit 0b63d25f604ed8c5320156b0846a1fdfa7553639
Author: Shuyi Chen 
Date:   2018-03-29T18:46:38Z

remove RexSimplify copy from calcite




---

[GitHub] flink pull request #5792: [Flink-8563][Table API & SQL] add unittest for con...

[suez1224]

GitHub user suez1224 opened a pull request:

https://github.com/apache/flink/pull/5792

[Flink-8563][Table API & SQL] add unittest for consecutive dot access of 
composite array element in SQL

 ## What is the purpose of the change

add unittest for consecutive dot access of composite array element in 
SQL. This depends on https://github.com/apache/flink/pull/5791.


## Brief change log

  -  add unittest


## Verifying this change

This change is already covered by existing tests.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (y no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/suez1224/flink FLINK-8563

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5792.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5792


commit 637b7462ac049d347fb5bbb8e68ed4fca7b0ec93
Author: Shuyi Chen 
Date:   2018-03-29T18:39:09Z

upgrade calcite dependency to 1.16

commit aea021cb9efc869872595f64467f8d2ec8071ea4
Author: Shuyi Chen 
Date:   2018-03-29T19:15:15Z

add unittest for consecutive dot access of composite array element in SQL




---

[jira] [Commented] (FLINK-8507) Upgrade Calcite dependency to 1.16

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419553#comment-16419553
 ] 

ASF GitHub Bot commented on FLINK-8507:
---

GitHub user suez1224 opened a pull request:

https://github.com/apache/flink/pull/5791

[FLINK-8507][Table API & SQL] upgrade calcite dependency to 1.16


## What is the purpose of the change

Upgrade Flink table's Calcite dependency to 1.16. 


## Brief change log

  - Update pom.
  - Fix HepPlanner issue.


## Verifying this change

This change is already covered by existing tests, such as *(please describe 
tests)*.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (y no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/suez1224/flink FLINK-8507

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5791.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5791


commit 637b7462ac049d347fb5bbb8e68ed4fca7b0ec93
Author: Shuyi Chen 
Date:   2018-03-29T18:39:09Z

upgrade calcite dependency to 1.16




> Upgrade Calcite dependency to 1.16
> --
>
> Key: FLINK-8507
> URL: https://issues.apache.org/jira/browse/FLINK-8507
> Project: Flink
>  Issue Type: Task
>  Components: Table API  SQL
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5791: [FLINK-8507][Table API & SQL] upgrade calcite depe...

[suez1224]

GitHub user suez1224 opened a pull request:

https://github.com/apache/flink/pull/5791

[FLINK-8507][Table API & SQL] upgrade calcite dependency to 1.16


## What is the purpose of the change

Upgrade Flink table's Calcite dependency to 1.16. 


## Brief change log

  - Update pom.
  - Fix HepPlanner issue.


## Verifying this change

This change is already covered by existing tests, such as *(please describe 
tests)*.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (y no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/suez1224/flink FLINK-8507

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5791.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5791


commit 637b7462ac049d347fb5bbb8e68ed4fca7b0ec93
Author: Shuyi Chen 
Date:   2018-03-29T18:39:09Z

upgrade calcite dependency to 1.16




---

[jira] [Commented] (FLINK-9111) SSL Passwords written to log file in plain text

[Vinay (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419474#comment-16419474
 ] 

Vinay commented on FLINK-9111:
--

[~aljoscha] that's right . sorry I thought this issue was not reported earlier.

> SSL Passwords written to log file in plain text
> ---
>
> Key: FLINK-9111
> URL: https://issues.apache.org/jira/browse/FLINK-9111
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.4.2
>Reporter: Vinay
>Priority: Major
>
> Hi,
> The SSL passwords are written to log file in plain text.
> This should be either be masked or should not be included in logs. 
> GlobalConfiguration file prints all the key and value in loadYAMLResource 
> method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9080) Flink Scheduler goes OOM, suspecting a memory leak

[Rohit Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419305#comment-16419305
 ] 

Rohit Singh commented on FLINK-9080:


Hi Till,

Tried 1.4.2 flink, but not 1.5.0

Posted this on stackoverflow 
https://stackoverflow.com/questions/49530333/getting-following-class-cast-exception-while-adding-job-jar-to-flink-home-lib

Based on that found out that, Our job contains  compile group: 
'org.apache.commons', name: 'commons-collections4', version: '4.1' as 
dependency 

and flink uses   compile group: 'commons-collections', name: 
'commons-collections', version: '3.2.2'  i. e 3.2.2 version removed the 
dependency and use the same dependecy whcih flink uses, but still was getting 
the same error. 



I can try out 1.5.0 release branch fix andd share the results, is there any fix 
targeted around this issue. Also in long term, is there any plan to avoid 
dynamic class loading in mesos or any other workaround to overcome the issue 
apart from adding jar in flink lib. Please let me know your thoughts on this.

 

 

> Flink Scheduler goes OOM, suspecting a memory leak
> --
>
> Key: FLINK-9080
> URL: https://issues.apache.org/jira/browse/FLINK-9080
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.4.0
>Reporter: Rohit Singh
>Priority: Blocker
> Fix For: 1.5.0
>
> Attachments: Top Level packages.JPG, Top level classes.JPG, 
> classesloaded vs unloaded.png
>
>
> Running FLink version 1.4.0. on mesos,scheduler running along  with job 
> manager in single container, whereas task managers running in seperate 
> containers.
> Couple of jobs were running continously, Flink scheduler was working 
> properlyalong with task managers. Due to some change in data, one of the jobs 
> started failing continuously. In the meantime,there was a surge in  flink 
> scheduler memory usually eventually died out off OOM
>  
> Memory dump analysis was done, 
> Following were findings  !Top Level packages.JPG!!Top level classes.JPG!
>  *  Majority of top loaded packages retaining heap indicated towards 
> Flinkuserclassloader, glassfish(jersey library), Finalizer classes. (Top 
> level package image)
>  * Top level classes were of Flinkuserclassloader, (Top Level class image)
>  * The number of classes loaded vs unloaded was quite less  PFA,inspite of 
> adding jvm options of -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled , 
> PFAclassloaded vs unloaded graph, scheduler was restarted 3 times
>  * There were custom classes as well which were duplicated during subsequent 
> class uploads
> PFA all the images of heap dump.  Can you suggest some pointers on as to how 
> to overcome this issue.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5790: [FLINK-9107][docs] document timer coalescing for ProcessF...

[NicoK]

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/5790
  
Actually, more advanced schemes using `current watermark + 1` (which fires 
with the next watermark) for the event time timer should also go into the 
documentation. I'll extend the PR ...


---

[jira] [Commented] (FLINK-9107) Document timer coalescing for ProcessFunctions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419302#comment-16419302
 ] 

ASF GitHub Bot commented on FLINK-9107:
---

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/5790
  
Actually, more advanced schemes using `current watermark + 1` (which fires 
with the next watermark) for the event time timer should also go into the 
documentation. I'll extend the PR ...


> Document timer coalescing for ProcessFunctions
> --
>
> Key: FLINK-9107
> URL: https://issues.apache.org/jira/browse/FLINK-9107
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation, Streaming
>Affects Versions: 1.3.0, 1.4.0, 1.5.0, 1.6.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Fix For: 1.5.0, 1.4.3, 1.3.4
>
>
> In a {{ProcessFunction}}, registering timers for each event via 
> {{ctx.timerService().registerEventTimeTimer()}} using times like 
> {{ctx.timestamp() + timeout}} will get a millisecond accuracy and may thus 
> create one timer per millisecond which may lead to some overhead in the 
> {{TimerService}}.
> This problem can be mitigated by using timer coalescing if the desired 
> accuracy of the timer can be larger than 1ms. A timer firing at full seconds 
> only, for example, can be realised like this:
> {code}
> coalescedTime = ((ctx.timestamp() + timeout) / 1000) * 1000;
> ctx.timerService().registerEventTimeTimer(coalescedTime);
> {code}
> As a result, only a single timer may exist for every second since we do not 
> add timers for timestamps that are already there.
> This should be documented in the {{ProcessFunction}} docs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9113) Data loss in BucketingSink when writing to local filesystem

[Timo Walther (JIRA)]

Timo Walther created FLINK-9113:
---

 Summary: Data loss in BucketingSink when writing to local 
filesystem
 Key: FLINK-9113
 URL: https://issues.apache.org/jira/browse/FLINK-9113
 Project: Flink
  Issue Type: Bug
  Components: Streaming Connectors
Reporter: Timo Walther
Assignee: Timo Walther


This issue is closely related to FLINK-7737. By default the bucketing sink uses 
HDFS's {{org.apache.hadoop.fs.FSDataOutputStream#hflush}} for performance 
reasons. However, this leads to data loss in case of TaskManager failures when 
writing to a local filesystem {{org.apache.hadoop.fs.LocalFileSystem}}. We 
should use {{hsync}} by default in local filesystem cases and make it possible 
to disable this behavior if needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8908) MapSerializer creates new serializer even if key and value serializers are stateless

[Kostas Kloudas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostas Kloudas closed FLINK-8908.
-
Resolution: Fixed

Fixed on master with eea887b7d9a3cd43416feca568c9815d8362e8d4

and on release-1.5 with d4a2bc545a1e993130479813d4194727a56d86f9

> MapSerializer creates new serializer even if key and value serializers are 
> stateless
> 
>
> Key: FLINK-8908
> URL: https://issues.apache.org/jira/browse/FLINK-8908
> Project: Flink
>  Issue Type: Bug
>  Components: Type Serialization System
>Affects Versions: 1.5.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>Priority: Major
> Fix For: 1.5.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8928) Improve error message on server binding error.

[Kostas Kloudas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostas Kloudas closed FLINK-8928.
-
Resolution: Fixed

Fixed on master with 6a8172a9c654589faa01f1fccb2dec5e008fe532

and on release-1.5 with c342487466abc70447550e05e2abe30e9f01e368

> Improve error message on server binding error.
> --
>
> Key: FLINK-8928
> URL: https://issues.apache.org/jira/browse/FLINK-8928
> Project: Flink
>  Issue Type: Bug
>  Components: Queryable State
>Affects Versions: 1.5.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>Priority: Major
> Fix For: 1.5.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-9097) Jobs can be dropped in HA when job submission fails

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-9097.

Resolution: Fixed

Fixed via
master: cc190a6457cdf6186ea8e449f7b38be04761af8b
1.5.0: 74f4d55be1edbb2fcbd25bff89de3d6ee790fea5

> Jobs can be dropped in HA when job submission fails
> ---
>
> Key: FLINK-9097
> URL: https://issues.apache.org/jira/browse/FLINK-9097
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Jobs can be dropped in HA mode if the job submission step fails. In such a 
> case, we should fail fatally to let the {{Dispatcher}} restart and retry to 
> recover all jobs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9097) Jobs can be dropped in HA when job submission fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419265#comment-16419265
 ] 

ASF GitHub Bot commented on FLINK-9097:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5774


> Jobs can be dropped in HA when job submission fails
> ---
>
> Key: FLINK-9097
> URL: https://issues.apache.org/jira/browse/FLINK-9097
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Jobs can be dropped in HA mode if the job submission step fails. In such a 
> case, we should fail fatally to let the {{Dispatcher}} restart and retry to 
> recover all jobs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8802) Concurrent serialization without duplicating serializers in state server.

[Kostas Kloudas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostas Kloudas closed FLINK-8802.
-
Resolution: Fixed

Fixed on master with db8e1f09bd7dcd9f392bf987e96cddcb34665b6c

and on release-1.5 with c17c3b60381b454e101d10b5b285b489775bfa71

> Concurrent serialization without duplicating serializers in state server.
> -
>
> Key: FLINK-8802
> URL: https://issues.apache.org/jira/browse/FLINK-8802
> Project: Flink
>  Issue Type: Bug
>  Components: Queryable State
>Affects Versions: 1.5.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The `getSerializedValue()` may be called by multiple threads but serializers 
> are not duplicated, which may lead to exceptions thrown when a serializer is 
> stateful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9106) Add UnfencedMainThreadExecutor to FencedRpcEndpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419264#comment-16419264
 ] 

ASF GitHub Bot commented on FLINK-9106:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5784


> Add UnfencedMainThreadExecutor to FencedRpcEndpoint
> ---
>
> Key: FLINK-9106
> URL: https://issues.apache.org/jira/browse/FLINK-9106
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> In order to run unfenced operation it would be convenient to also have an 
> {{UnfencedMainThreadExecutor}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5774: [FLINK-9097] Fail fatally if job submission fails ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5774


---

[GitHub] flink pull request #5784: [FLINK-9106] [rpc] Add UnfencedMainThreadExecutor ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5784


---

[jira] [Closed] (FLINK-9012) Shaded Hadoop S3A end-to-end test failing with S3 bucket in Ireland

[Nico Kruber (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nico Kruber closed FLINK-9012.
--
   Resolution: Cannot Reproduce
Fix Version/s: (was: 1.5.0)

Yes, I now tested locally and it was running fine with the same S3 settings. 
Maybe a temporary glitch with S3.

> Shaded Hadoop S3A end-to-end test failing with S3 bucket in Ireland
> ---
>
> Key: FLINK-9012
> URL: https://issues.apache.org/jira/browse/FLINK-9012
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.5.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Critical
>  Labels: test-stability
>
> https://api.travis-ci.org/v3/job/354259892/log.txt
> {code}
> Found AWS bucket [secure], running Shaded Hadoop S3A e2e tests.
> Flink dist directory: /home/travis/build/NicoK/flink/build-target
> TEST_DATA_DIR: 
> /home/travis/build/NicoK/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-05775180416
>   % Total% Received % Xferd  Average Speed   TimeTime Time  
> Current
>  Dload  Upload   Total   SpentLeft  Speed
>   0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
>   0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
>  91   4930   4490 0   2476  0 --:--:-- --:--:-- --:--:--  2467
> 
> TemporaryRedirectPlease re-send this request to 
> the specified temporary endpoint. Continue to use the original request 
> endpoint for future 
> requests.[secure][secure].s3-eu-west-1.amazonaws.com1FCEC82C3EBF7C7ENG5dxnXQ0Y5mK2X/m3bU+Z7Fqt0mNVL2JsjyVjGZUmpDmNuBDfKJACh7VI6tCTYEZsw65W057lA=Starting
>  cluster.
> Starting standalonesession daemon on host 
> travis-job-087822e3-2f4c-46b7-b9bd-b6d4aba6136c.
> Starting taskexecutor daemon on host 
> travis-job-087822e3-2f4c-46b7-b9bd-b6d4aba6136c.
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher REST endpoint is up.
> Starting execution of program
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: Could not 
> retrieve the execution result.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:246)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:458)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:446)
>   at 
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
>   at 
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:86)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:398)
>   at 
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:780)
>   at 
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:274)
>   at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:209)
>   at 
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1019)
>   at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1095)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>   at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1095)
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to 
> submit JobGraph.
>   

[jira] [Commented] (FLINK-8802) Concurrent serialization without duplicating serializers in state server.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419248#comment-16419248
 ] 

ASF GitHub Bot commented on FLINK-8802:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5691


> Concurrent serialization without duplicating serializers in state server.
> -
>
> Key: FLINK-8802
> URL: https://issues.apache.org/jira/browse/FLINK-8802
> Project: Flink
>  Issue Type: Bug
>  Components: Queryable State
>Affects Versions: 1.5.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The `getSerializedValue()` may be called by multiple threads but serializers 
> are not duplicated, which may lead to exceptions thrown when a serializer is 
> stateful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9080) Flink Scheduler goes OOM, suspecting a memory leak

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419249#comment-16419249
 ] 

Till Rohrmann commented on FLINK-9080:
--

Hi [~rohsing], have you tried whether the problem also occurs with the Flink 
1.5 release branch?

> Flink Scheduler goes OOM, suspecting a memory leak
> --
>
> Key: FLINK-9080
> URL: https://issues.apache.org/jira/browse/FLINK-9080
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.4.0
>Reporter: Rohit Singh
>Priority: Blocker
> Fix For: 1.5.0
>
> Attachments: Top Level packages.JPG, Top level classes.JPG, 
> classesloaded vs unloaded.png
>
>
> Running FLink version 1.4.0. on mesos,scheduler running along  with job 
> manager in single container, whereas task managers running in seperate 
> containers.
> Couple of jobs were running continously, Flink scheduler was working 
> properlyalong with task managers. Due to some change in data, one of the jobs 
> started failing continuously. In the meantime,there was a surge in  flink 
> scheduler memory usually eventually died out off OOM
>  
> Memory dump analysis was done, 
> Following were findings  !Top Level packages.JPG!!Top level classes.JPG!
>  *  Majority of top loaded packages retaining heap indicated towards 
> Flinkuserclassloader, glassfish(jersey library), Finalizer classes. (Top 
> level package image)
>  * Top level classes were of Flinkuserclassloader, (Top Level class image)
>  * The number of classes loaded vs unloaded was quite less  PFA,inspite of 
> adding jvm options of -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled , 
> PFAclassloaded vs unloaded graph, scheduler was restarted 3 times
>  * There were custom classes as well which were duplicated during subsequent 
> class uploads
> PFA all the images of heap dump.  Can you suggest some pointers on as to how 
> to overcome this issue.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5691: [FLINK-8802] [QS] Fix concurrent access to non-dup...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5691


---

[jira] [Commented] (FLINK-9010) NoResourceAvailableException with FLIP-6

[Nico Kruber (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419245#comment-16419245
 ] 

Nico Kruber commented on FLINK-9010:


Sorry for the late response.
 * This distinction between "logical slots" and "physical slots" is confusing 
and I somehow doubt that this is documented (but maybe I'm wrong here). We 
should probably at least adapt the log messages for something that the user is 
expecting when he's reading "slots".
 * Regarding the actual issue: there should always have been enough machines to 
offer all needed slots but I cannot rule out that some EC2 instance was 
unresponsive or currently restarting due to some failure. I think, [~pnowojski] 
did some more experiments with big cluster setups recently and I haven't heard 
of that occurring again (correct me if I'm wrong). If it did not occur again, 
we may close this issue as it may have been an EC2 hickup.

> NoResourceAvailableException with FLIP-6 
> -
>
> Key: FLINK-9010
> URL: https://issues.apache.org/jira/browse/FLINK-9010
> Project: Flink
>  Issue Type: Bug
>  Components: ResourceManager
>Affects Versions: 1.5.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> I was trying to run a bigger program with 400 slots (100 TMs, 2 slots each) 
> with FLIP-6 mode and a checkpointing interval of 1000 and got the following 
> exception:
> {code}
> 2018-03-16 03:41:20,154 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - Received new container: 
> container_1521038088305_0257_01_000101 - Remaining pending container 
> requests: 302
> 2018-03-16 03:41:20,154 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - TaskExecutor container_1521038088305_0257_01_000101 will be 
> started with container size 8192 MB, JVM heap size 5120 MB, JVM direct memory 
> limit 3072 MB
> 2018-03-16 03:41:20,154 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - TM:remote keytab path obtained null
> 2018-03-16 03:41:20,154 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - TM:remote keytab principal obtained null
> 2018-03-16 03:41:20,154 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - TM:remote yarn conf path obtained null
> 2018-03-16 03:41:20,154 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - TM:remote krb5 path obtained null
> 2018-03-16 03:41:20,155 INFO  org.apache.flink.yarn.Utils 
>   - Copying from 
> file:/mnt/yarn/usercache/hadoop/appcache/application_1521038088305_0257/container_1521038088305_0257_01_01/3cd0c7d7-ccc1-4680-83a5-54e64dd637bc-taskmanager-conf.yaml
>  to 
> hdfs://ip-172-31-1-91.eu-west-1.compute.internal:8020/user/hadoop/.flink/application_1521038088305_0257/3cd0c7d7-ccc1-4680-83a5-54e64dd637bc-taskmanager-conf.yaml
> 2018-03-16 03:41:20,165 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - Prepared local resource for modified yaml: resource { scheme: 
> "hdfs" host: "ip-172-31-1-91.eu-west-1.compute.internal" port: 8020 file: 
> "/user/hadoop/.flink/application_1521038088305_0257/3cd0c7d7-ccc1-4680-83a5-54e64dd637bc-taskmanager-conf.yaml"
>  } size: 595 timestamp: 1521171680164 type: FILE visibility: APPLICATION
> 2018-03-16 03:41:20,168 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - Creating container launch context for TaskManagers
> 2018-03-16 03:41:20,168 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - Starting TaskManagers with command: $JAVA_HOME/bin/java 
> -Xms5120m -Xmx5120m -XX:MaxDirectMemorySize=3072m  
> -Dlog.file=/taskmanager.log 
> -Dlogback.configurationFile=file:./logback.xml 
> -Dlog4j.configuration=file:./log4j.properties 
> org.apache.flink.yarn.YarnTaskExecutorRunner --configDir . 1> 
> /taskmanager.out 2> /taskmanager.err
> 2018-03-16 03:41:20,176 INFO  
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy  - 
> Opening proxy : ip-172-31-3-221.eu-west-1.compute.internal:8041
> 2018-03-16 03:41:20,180 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - Received new container: 
> container_1521038088305_0257_01_000102 - Remaining pending container 
> requests: 301
> 2018-03-16 03:41:20,180 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - TaskExecutor container_1521038088305_0257_01_000102 will be 
> started with container size 8192 MB, JVM heap size 5120 MB, JVM direct memory 
> limit 3072 MB
> 2018-03-16 03:41:20,180 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - TM:remote keytab path obtained null
> 2018-03-16 03:41:20,180 INFO  

[jira] [Comment Edited] (FLINK-9105) Table program compiles failed

[Timo Walther (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419241#comment-16419241
 ] 

Timo Walther edited comment on FLINK-9105 at 3/29/18 3:57 PM:
--

[~bob365] thanks for opening this issue. Could you also show a reproducible SQL 
query to this error? At a first glance the NoSuchMethodError looks like a 
dependency issue of the project you are implementing. Make sure that your code 
does not override the Janino compiler of {{flink-table}}. Pleas let us know if 
this solved your issue.


was (Author: twalthr):
[~bob365] thanks for opening this issue. Could you also show a reproducible SQL 
query to this error? At a first glance the NoSuchMethodError looks like a 
dependency issue of the project you are implementing. Make sure that you code 
does not override the Janino compiler of {{flink-table}}. Pleas let us know if 
this solved your issue.

> Table program compiles failed
> -
>
> Key: FLINK-9105
> URL: https://issues.apache.org/jira/browse/FLINK-9105
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.4.0, 1.5.0, 1.4.1, 1.4.2
>Reporter: Bob Lau
>Priority: Major
>
> ExceptionStack:
> org.apache.flink.client.program.ProgramInvocationException: 
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be 
> compiled. This is a bug. Please file an issue.
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:253)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:463)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:456)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.executeRemotely(RemoteStreamEnvironment.java:219)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.execute(RemoteStreamEnvironment.java:178)
> at 
> com.tydic.tysc.job.service.SubmitJobService.submitJobToStandaloneCluster(SubmitJobService.java:150)
> at 
> com.tydic.tysc.rest.SubmitJobController.submitJob(SubmitJobController.java:55)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
> at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> 

[jira] [Commented] (FLINK-9105) Table program compiles failed

[Timo Walther (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419241#comment-16419241
 ] 

Timo Walther commented on FLINK-9105:
-

[~bob365] thanks for opening this issue. Could you also show a reproducible SQL 
query to this error? At a first glance the NoSuchMethodError looks like a 
dependency issue of the project you are implementing. Make sure that you code 
does not override the Janino compiler of {{flink-table}}. Pleas let us know if 
this solved your issue.

> Table program compiles failed
> -
>
> Key: FLINK-9105
> URL: https://issues.apache.org/jira/browse/FLINK-9105
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.4.0, 1.5.0, 1.4.1, 1.4.2
>Reporter: Bob Lau
>Priority: Major
>
> ExceptionStack:
> org.apache.flink.client.program.ProgramInvocationException: 
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be 
> compiled. This is a bug. Please file an issue.
> at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:253)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:463)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:456)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.executeRemotely(RemoteStreamEnvironment.java:219)
> at 
> org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.execute(RemoteStreamEnvironment.java:178)
> at 
> com.tydic.tysc.job.service.SubmitJobService.submitJobToStandaloneCluster(SubmitJobService.java:150)
> at 
> com.tydic.tysc.rest.SubmitJobController.submitJob(SubmitJobController.java:55)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
> at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> com.tydic.tysc.filter.ShiroSessionFilter.doFilter(ShiroSessionFilter.java:51)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> 

[jira] [Created] (FLINK-9112) Let end-to-end tests upload the components logs in case of a failure

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-9112:


 Summary: Let end-to-end tests upload the components logs in case 
of a failure
 Key: FLINK-9112
 URL: https://issues.apache.org/jira/browse/FLINK-9112
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Affects Versions: 1.5.0
Reporter: Till Rohrmann


I think it would quite helpful if the end-to-end tests which run as part of a 
Flink build would upload the component logs in case of a failure similar to how 
we do it for the maven tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-9012) Shaded Hadoop S3A end-to-end test failing with S3 bucket in Ireland

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-9012:
-
Priority: Critical  (was: Blocker)

> Shaded Hadoop S3A end-to-end test failing with S3 bucket in Ireland
> ---
>
> Key: FLINK-9012
> URL: https://issues.apache.org/jira/browse/FLINK-9012
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.5.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.5.0
>
>
> https://api.travis-ci.org/v3/job/354259892/log.txt
> {code}
> Found AWS bucket [secure], running Shaded Hadoop S3A e2e tests.
> Flink dist directory: /home/travis/build/NicoK/flink/build-target
> TEST_DATA_DIR: 
> /home/travis/build/NicoK/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-05775180416
>   % Total% Received % Xferd  Average Speed   TimeTime Time  
> Current
>  Dload  Upload   Total   SpentLeft  Speed
>   0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
>   0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
>  91   4930   4490 0   2476  0 --:--:-- --:--:-- --:--:--  2467
> 
> TemporaryRedirectPlease re-send this request to 
> the specified temporary endpoint. Continue to use the original request 
> endpoint for future 
> requests.[secure][secure].s3-eu-west-1.amazonaws.com1FCEC82C3EBF7C7ENG5dxnXQ0Y5mK2X/m3bU+Z7Fqt0mNVL2JsjyVjGZUmpDmNuBDfKJACh7VI6tCTYEZsw65W057lA=Starting
>  cluster.
> Starting standalonesession daemon on host 
> travis-job-087822e3-2f4c-46b7-b9bd-b6d4aba6136c.
> Starting taskexecutor daemon on host 
> travis-job-087822e3-2f4c-46b7-b9bd-b6d4aba6136c.
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher REST endpoint is up.
> Starting execution of program
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: Could not 
> retrieve the execution result.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:246)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:458)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:446)
>   at 
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
>   at 
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:86)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:398)
>   at 
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:780)
>   at 
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:274)
>   at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:209)
>   at 
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1019)
>   at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1095)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>   at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1095)
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to 
> submit JobGraph.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$4(RestClusterClient.java:341)
>  

[jira] [Commented] (FLINK-9012) Shaded Hadoop S3A end-to-end test failing with S3 bucket in Ireland

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419236#comment-16419236
 ] 

Till Rohrmann commented on FLINK-9012:
--

The problem is that the end-to-end tests don't upload the log files. Therefore, 
it is really hard to diagnose the problem. I would suggest to change the e2e 
tests such that they upload the logs and close this issue until we see another 
occurrence.

> Shaded Hadoop S3A end-to-end test failing with S3 bucket in Ireland
> ---
>
> Key: FLINK-9012
> URL: https://issues.apache.org/jira/browse/FLINK-9012
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.5.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.5.0
>
>
> https://api.travis-ci.org/v3/job/354259892/log.txt
> {code}
> Found AWS bucket [secure], running Shaded Hadoop S3A e2e tests.
> Flink dist directory: /home/travis/build/NicoK/flink/build-target
> TEST_DATA_DIR: 
> /home/travis/build/NicoK/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-05775180416
>   % Total% Received % Xferd  Average Speed   TimeTime Time  
> Current
>  Dload  Upload   Total   SpentLeft  Speed
>   0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
>   0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
>  91   4930   4490 0   2476  0 --:--:-- --:--:-- --:--:--  2467
> 
> TemporaryRedirectPlease re-send this request to 
> the specified temporary endpoint. Continue to use the original request 
> endpoint for future 
> requests.[secure][secure].s3-eu-west-1.amazonaws.com1FCEC82C3EBF7C7ENG5dxnXQ0Y5mK2X/m3bU+Z7Fqt0mNVL2JsjyVjGZUmpDmNuBDfKJACh7VI6tCTYEZsw65W057lA=Starting
>  cluster.
> Starting standalonesession daemon on host 
> travis-job-087822e3-2f4c-46b7-b9bd-b6d4aba6136c.
> Starting taskexecutor daemon on host 
> travis-job-087822e3-2f4c-46b7-b9bd-b6d4aba6136c.
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher REST endpoint is up.
> Starting execution of program
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: Could not 
> retrieve the execution result.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:246)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:458)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:446)
>   at 
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
>   at 
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:86)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:398)
>   at 
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:780)
>   at 
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:274)
>   at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:209)
>   at 
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1019)
>   at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1095)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>   at 

[GitHub] flink pull request #5790: [FLINK-9107][docs] document timer coalescing for P...

[NicoK]

GitHub user NicoK opened a pull request:

https://github.com/apache/flink/pull/5790

[FLINK-9107][docs] document timer coalescing for ProcessFunction

## What is the purpose of the change

In a ProcessFunction, registering timers for each event via 
`ctx.timerService().registerEventTimeTimer()` using timestamps like 
`ctx.timestamp() + timeout` will get a millisecond accuracy and may thus create 
one timer per millisecond which may lead to some overhead in the `TimerService`.

This problem can be mitigated by using timer coalescing if the desired 
accuracy of the timer can be larger than 1ms. A timer firing at full seconds 
only, for example, can be realised like this:

```
long coalescedTime = ((ctx.timestamp() + timeout) / 1000) * 1000;
ctx.timerService().registerEventTimeTimer(coalescedTime);
```

As a result, only a single timer may exist for every second since we do not 
add timers for timestamps that are already there.

Please note that this PR includes #5788 and should also be merged into 1.3 
and 1.4 docs to which it applies as well.

## Brief change log

- document timer coalescing for `ProcessFunction`


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NicoK/flink flink-9107

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5790.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5790


commit b6258697aabb8688aac19b5b228ddf8926e518f3
Author: Nico Kruber 
Date:   2018-03-29T13:55:47Z

[FLINK-9110][docs] fix local bundler installation

commit 540485a5620f6dcdd98e751a086076fb80997f65
Author: Nico Kruber 
Date:   2018-03-29T13:56:23Z

[hotfix][docs] remove duplicate bundle installation

commit f8274cf3ba06af19a4ac31e784803723e449336a
Author: Nico Kruber 
Date:   2018-03-29T14:20:00Z

[FLINK-9107][docs] document timer coalescing for ProcessFunction




---

[jira] [Commented] (FLINK-9107) Document timer coalescing for ProcessFunctions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419233#comment-16419233
 ] 

ASF GitHub Bot commented on FLINK-9107:
---

GitHub user NicoK opened a pull request:

https://github.com/apache/flink/pull/5790

[FLINK-9107][docs] document timer coalescing for ProcessFunction

## What is the purpose of the change

In a ProcessFunction, registering timers for each event via 
`ctx.timerService().registerEventTimeTimer()` using timestamps like 
`ctx.timestamp() + timeout` will get a millisecond accuracy and may thus create 
one timer per millisecond which may lead to some overhead in the `TimerService`.

This problem can be mitigated by using timer coalescing if the desired 
accuracy of the timer can be larger than 1ms. A timer firing at full seconds 
only, for example, can be realised like this:

```
long coalescedTime = ((ctx.timestamp() + timeout) / 1000) * 1000;
ctx.timerService().registerEventTimeTimer(coalescedTime);
```

As a result, only a single timer may exist for every second since we do not 
add timers for timestamps that are already there.

Please note that this PR includes #5788 and should also be merged into 1.3 
and 1.4 docs to which it applies as well.

## Brief change log

- document timer coalescing for `ProcessFunction`


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NicoK/flink flink-9107

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5790.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5790


commit b6258697aabb8688aac19b5b228ddf8926e518f3
Author: Nico Kruber 
Date:   2018-03-29T13:55:47Z

[FLINK-9110][docs] fix local bundler installation

commit 540485a5620f6dcdd98e751a086076fb80997f65
Author: Nico Kruber 
Date:   2018-03-29T13:56:23Z

[hotfix][docs] remove duplicate bundle installation

commit f8274cf3ba06af19a4ac31e784803723e449336a
Author: Nico Kruber 
Date:   2018-03-29T14:20:00Z

[FLINK-9107][docs] document timer coalescing for ProcessFunction




> Document timer coalescing for ProcessFunctions
> --
>
> Key: FLINK-9107
> URL: https://issues.apache.org/jira/browse/FLINK-9107
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation, Streaming
>Affects Versions: 1.3.0, 1.4.0, 1.5.0, 1.6.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Fix For: 1.5.0, 1.4.3, 1.3.4
>
>
> In a {{ProcessFunction}}, registering timers for each event via 
> {{ctx.timerService().registerEventTimeTimer()}} using times like 
> {{ctx.timestamp() + timeout}} will get a millisecond accuracy and may thus 
> create one timer per millisecond which may lead to some overhead in the 
> {{TimerService}}.
> This problem can be mitigated by using timer coalescing if the desired 
> accuracy of the timer can be larger than 1ms. A timer firing at full seconds 
> only, for example, can be realised like this:
> {code}
> coalescedTime = ((ctx.timestamp() + timeout) / 1000) * 1000;
> ctx.timerService().registerEventTimeTimer(coalescedTime);
> {code}
> As a result, only a single timer may exist for every second since we do not 
> add timers for timestamps that are already there.
> This should be documented in the {{ProcessFunction}} docs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8897) Rowtime materialization causes "mismatched type" AssertionError

[Timo Walther (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419231#comment-16419231
 ] 

Timo Walther commented on FLINK-8897:
-

[~till.rohrmann] it has been introduced with Flink 1.5 but it must not be a 
blocker.

> Rowtime materialization causes "mismatched type" AssertionError
> ---
>
> Key: FLINK-8897
> URL: https://issues.apache.org/jira/browse/FLINK-8897
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Reporter: Xingcan Cui
>Assignee: Timo Walther
>Priority: Major
> Fix For: 1.5.0
>
>
> As raised in [this 
> thread|https://lists.apache.org/thread.html/e2ea38aa7ae224d7481145334955d84243690e9aad10d58310bdb8e7@%3Cuser.flink.apache.org%3E],
>  the query created by the following code will throw a calcite "mismatch type" 
> ({{Timestamp(3)}} and {{TimeIndicator}}) exception.
> {code:java}
> String sql1 = "select id, eventTs as t1, count(*) over (partition by id order 
> by eventTs rows between 100 preceding and current row) as cnt1 from myTable1";
> String sql2 = "select distinct id as r_id, eventTs as t2, count(*) over 
> (partition by id order by eventTs rows between 50 preceding and current row) 
> as cnt2 from myTable2";
> Table left = tableEnv.sqlQuery(sql1);
> Table right = tableEnv.sqlQuery(sql2);
> left.join(right).where("id === r_id && t1 === t2").select("id, 
> t1").writeToSink(...)
> {code}
> The logical plan is as follows.
> {code}
> LogicalProject(id=[$0], t1=[$1])
>   LogicalFilter(condition=[AND(=($0, $3), =($1, $4))])
> LogicalJoin(condition=[true], joinType=[inner])
>   LogicalAggregate(group=[{0, 1, 2}])
> LogicalWindow(window#0=[window(partition {0} order by [1] rows 
> between $2 PRECEDING and CURRENT ROW aggs [COUNT()])])
>   LogicalProject(id=[$0], eventTs=[$3])
> LogicalTableScan(table=[[_DataStreamTable_0]])
>   LogicalAggregate(group=[{0, 1, 2}])
> LogicalWindow(window#0=[window(partition {0} order by [1] rows 
> between $2 PRECEDING and CURRENT ROW aggs [COUNT()])])
>   LogicalProject(id=[$0], eventTs=[$3])
> LogicalTableScan(table=[[_DataStreamTable_0]])
> {code}
> That is because the the rowtime field after an aggregation will be 
> materialized while the {{RexInputRef}} type for the filter's operands ({{t1 
> === t2}}) is still {{TimeIndicator}}. We should make them unified.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8897) Rowtime materialization causes "mismatched type" AssertionError

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther updated FLINK-8897:

Priority: Major  (was: Blocker)

> Rowtime materialization causes "mismatched type" AssertionError
> ---
>
> Key: FLINK-8897
> URL: https://issues.apache.org/jira/browse/FLINK-8897
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Reporter: Xingcan Cui
>Assignee: Timo Walther
>Priority: Major
> Fix For: 1.5.0
>
>
> As raised in [this 
> thread|https://lists.apache.org/thread.html/e2ea38aa7ae224d7481145334955d84243690e9aad10d58310bdb8e7@%3Cuser.flink.apache.org%3E],
>  the query created by the following code will throw a calcite "mismatch type" 
> ({{Timestamp(3)}} and {{TimeIndicator}}) exception.
> {code:java}
> String sql1 = "select id, eventTs as t1, count(*) over (partition by id order 
> by eventTs rows between 100 preceding and current row) as cnt1 from myTable1";
> String sql2 = "select distinct id as r_id, eventTs as t2, count(*) over 
> (partition by id order by eventTs rows between 50 preceding and current row) 
> as cnt2 from myTable2";
> Table left = tableEnv.sqlQuery(sql1);
> Table right = tableEnv.sqlQuery(sql2);
> left.join(right).where("id === r_id && t1 === t2").select("id, 
> t1").writeToSink(...)
> {code}
> The logical plan is as follows.
> {code}
> LogicalProject(id=[$0], t1=[$1])
>   LogicalFilter(condition=[AND(=($0, $3), =($1, $4))])
> LogicalJoin(condition=[true], joinType=[inner])
>   LogicalAggregate(group=[{0, 1, 2}])
> LogicalWindow(window#0=[window(partition {0} order by [1] rows 
> between $2 PRECEDING and CURRENT ROW aggs [COUNT()])])
>   LogicalProject(id=[$0], eventTs=[$3])
> LogicalTableScan(table=[[_DataStreamTable_0]])
>   LogicalAggregate(group=[{0, 1, 2}])
> LogicalWindow(window#0=[window(partition {0} order by [1] rows 
> between $2 PRECEDING and CURRENT ROW aggs [COUNT()])])
>   LogicalProject(id=[$0], eventTs=[$3])
> LogicalTableScan(table=[[_DataStreamTable_0]])
> {code}
> That is because the the rowtime field after an aggregation will be 
> materialized while the {{RexInputRef}} type for the filter's operands ({{t1 
> === t2}}) is still {{TimeIndicator}}. We should make them unified.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-9012) Shaded Hadoop S3A end-to-end test failing with S3 bucket in Ireland

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-9012:
-
Labels: test-stability  (was: )

> Shaded Hadoop S3A end-to-end test failing with S3 bucket in Ireland
> ---
>
> Key: FLINK-9012
> URL: https://issues.apache.org/jira/browse/FLINK-9012
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.5.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.5.0
>
>
> https://api.travis-ci.org/v3/job/354259892/log.txt
> {code}
> Found AWS bucket [secure], running Shaded Hadoop S3A e2e tests.
> Flink dist directory: /home/travis/build/NicoK/flink/build-target
> TEST_DATA_DIR: 
> /home/travis/build/NicoK/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-05775180416
>   % Total% Received % Xferd  Average Speed   TimeTime Time  
> Current
>  Dload  Upload   Total   SpentLeft  Speed
>   0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
>   0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
>  91   4930   4490 0   2476  0 --:--:-- --:--:-- --:--:--  2467
> 
> TemporaryRedirectPlease re-send this request to 
> the specified temporary endpoint. Continue to use the original request 
> endpoint for future 
> requests.[secure][secure].s3-eu-west-1.amazonaws.com1FCEC82C3EBF7C7ENG5dxnXQ0Y5mK2X/m3bU+Z7Fqt0mNVL2JsjyVjGZUmpDmNuBDfKJACh7VI6tCTYEZsw65W057lA=Starting
>  cluster.
> Starting standalonesession daemon on host 
> travis-job-087822e3-2f4c-46b7-b9bd-b6d4aba6136c.
> Starting taskexecutor daemon on host 
> travis-job-087822e3-2f4c-46b7-b9bd-b6d4aba6136c.
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher/TaskManagers are not yet up
> Waiting for dispatcher REST endpoint to come up...
> Waiting for dispatcher REST endpoint to come up...
> Dispatcher REST endpoint is up.
> Starting execution of program
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: Could not 
> retrieve the execution result.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:246)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:458)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:446)
>   at 
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
>   at 
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:86)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:398)
>   at 
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:780)
>   at 
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:274)
>   at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:209)
>   at 
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1019)
>   at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1095)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>   at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1095)
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to 
> submit JobGraph.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$4(RestClusterClient.java:341)
>  

[jira] [Closed] (FLINK-9065) CorrelateITCase failed on travis

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-9065.

Resolution: Fixed

Hopefully fix by: b0fbb89898bdfc72445464a2547e088e825f4c55

> CorrelateITCase failed on travis
> 
>
> Key: FLINK-9065
> URL: https://issues.apache.org/jira/browse/FLINK-9065
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager, Table API  SQL
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> https://travis-ci.org/zentol/flink/jobs/357296815
> {code}
> (org.apache.flink.table.runtime.batch.table.CorrelateITCase)  Time elapsed: 
> 0.157 sec  <<< ERROR!
> org.apache.flink.runtime.client.JobExecutionException: Could not retrieve 
> JobResult.
>   at 
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:619)
>   at 
> org.apache.flink.test.util.TestEnvironment.execute(TestEnvironment.java:120)
>   at 
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:815)
>   at 
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:526)
>   at org.apache.flink.api.scala.DataSet.collect(DataSet.scala:601)
>   at 
> org.apache.flink.table.runtime.batch.table.CorrelateITCase.testCrossJoin(CorrelateITCase.scala:64)
> Caused by: java.lang.IllegalArgumentException: netRuntime must be greater 
> than or equals 0
>   at 
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139)
>   at 
> org.apache.flink.runtime.jobmaster.JobResult.(JobResult.java:70)
>   at 
> org.apache.flink.runtime.jobmaster.JobResult.(JobResult.java:50)
>   at 
> org.apache.flink.runtime.jobmaster.JobResult$Builder.build(JobResult.java:165)
>   at 
> org.apache.flink.runtime.jobmaster.JobResult.createFrom(JobResult.java:202)
>   at 
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>   at 
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>   at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>   at 
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>   at 
> org.apache.flink.runtime.jobmaster.JobManagerRunner.jobReachedGloballyTerminalState(JobManagerRunner.java:248)
>   at 
> org.apache.flink.runtime.jobmaster.JobMaster.lambda$jobStatusChanged$14(JobMaster.java:1185)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9065) CorrelateITCase failed on travis

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419216#comment-16419216
 ] 

Till Rohrmann commented on FLINK-9065:
--

This could be related to a clock shift because we are using 
{{System.currentTimeMillis}} to assign the state timestamps. I've committed 
b0fbb89898bdfc72445464a2547e088e825f4c55 which hopefully guards against this 
problem.

> CorrelateITCase failed on travis
> 
>
> Key: FLINK-9065
> URL: https://issues.apache.org/jira/browse/FLINK-9065
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager, Table API  SQL
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> https://travis-ci.org/zentol/flink/jobs/357296815
> {code}
> (org.apache.flink.table.runtime.batch.table.CorrelateITCase)  Time elapsed: 
> 0.157 sec  <<< ERROR!
> org.apache.flink.runtime.client.JobExecutionException: Could not retrieve 
> JobResult.
>   at 
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:619)
>   at 
> org.apache.flink.test.util.TestEnvironment.execute(TestEnvironment.java:120)
>   at 
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:815)
>   at 
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:526)
>   at org.apache.flink.api.scala.DataSet.collect(DataSet.scala:601)
>   at 
> org.apache.flink.table.runtime.batch.table.CorrelateITCase.testCrossJoin(CorrelateITCase.scala:64)
> Caused by: java.lang.IllegalArgumentException: netRuntime must be greater 
> than or equals 0
>   at 
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139)
>   at 
> org.apache.flink.runtime.jobmaster.JobResult.(JobResult.java:70)
>   at 
> org.apache.flink.runtime.jobmaster.JobResult.(JobResult.java:50)
>   at 
> org.apache.flink.runtime.jobmaster.JobResult$Builder.build(JobResult.java:165)
>   at 
> org.apache.flink.runtime.jobmaster.JobResult.createFrom(JobResult.java:202)
>   at 
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>   at 
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>   at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>   at 
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>   at 
> org.apache.flink.runtime.jobmaster.JobManagerRunner.jobReachedGloballyTerminalState(JobManagerRunner.java:248)
>   at 
> org.apache.flink.runtime.jobmaster.JobMaster.lambda$jobStatusChanged$14(JobMaster.java:1185)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9103) SSL verification on TaskManager when parallelism > 1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419213#comment-16419213
 ] 

ASF GitHub Bot commented on FLINK-9103:
---

GitHub user EAlexRojas opened a pull request:

https://github.com/apache/flink/pull/5789

[FLINK-9103] Using CanonicalHostName instead of IP for SSL connection on 
NettyClient

## What is the purpose of the change

This pull request makes the NettyClient use the CanonicalHostName instead 
of the IP address for SSL communication. That way dynamic environments like 
kubernetes can be fully supported as certificates with wildcard DNS can be used.


## Brief change log

- Use CanonicalHostName instead of HostNameAddress to identify the server 
on the NettyClient


## Verifying this change

This change is already covered by existing tests, such as:

NettyClientServerSslTest (org.apache.flink.runtime.io.network.netty)
   - testValidSslConnection
   - testSslHandshakeError 

Also manually verified the change by running a 4 node kubernetes cluster 
with 1 JobManagers and 3 TaskManagers, using wildcard DNS certificates and 
executing a stateful streaming program with parallelism set to 2 and verifying 
that all nodes are able to communicate to each other successfully. 

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency):  no
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
  - The serializers: no
  - The runtime per-record code paths (performance sensitive): no
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
  - The S3 file system connector: no

## Documentation

  - Does this pull request introduce a new feature? no
  - If yes, how is the feature documented? not applicable


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/EAlexRojas/flink release-1.4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5789.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5789


commit 202672da7901fe7df912e6a057d6d0c29ccaf0fd
Author: EAlexRojas 
Date:   2018-03-29T14:01:24Z

Using CanonicalHostName instead of IP for SSL coonection on NettyClient




> SSL verification on TaskManager when parallelism > 1
> 
>
> Key: FLINK-9103
> URL: https://issues.apache.org/jira/browse/FLINK-9103
> Project: Flink
>  Issue Type: Bug
>  Components: Docker, Security
>Affects Versions: 1.4.0
>Reporter: Edward Rojas
>Priority: Major
> Attachments: job.log, task0.log
>
>
> In dynamic environments like Kubernetes, the SSL certificates can be 
> generated to use only the DNS addresses for validation of the identity of 
> servers, given that the IP can change eventually.
>  
> In this cases when executing Jobs with Parallelism set to 1, the SSL 
> validations are good and the Jobmanager can communicate with Task manager and 
> vice versa.
>  
> But with parallelism set to more than 1, SSL validation fails when Task 
> Managers communicate to each other as it seems to try to validate against IP 
> address:
> Caused by: java.security.cert.CertificateException: No subject alternative 
> names matching IP address 172.xx.xxx.xxx found 
> at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:168) 
> at sun.security.util.HostnameChecker.match(HostnameChecker.java:94) 
> at 
> sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
>  
> at 
> sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
>  
> at 
> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
>  
> at 
> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>  
> at 
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601)
>  
> ... 21 more 
>  
> From the logs, it seems the task managers register successfully its full 
> address to Netty, but still the IP is used.
>  
> Attached pertinent logs from JobManager and a TaskManager. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5789: [FLINK-9103] Using CanonicalHostName instead of IP...

[EAlexRojas]

GitHub user EAlexRojas opened a pull request:

https://github.com/apache/flink/pull/5789

[FLINK-9103] Using CanonicalHostName instead of IP for SSL connection on 
NettyClient

## What is the purpose of the change

This pull request makes the NettyClient use the CanonicalHostName instead 
of the IP address for SSL communication. That way dynamic environments like 
kubernetes can be fully supported as certificates with wildcard DNS can be used.


## Brief change log

- Use CanonicalHostName instead of HostNameAddress to identify the server 
on the NettyClient


## Verifying this change

This change is already covered by existing tests, such as:

NettyClientServerSslTest (org.apache.flink.runtime.io.network.netty)
   - testValidSslConnection
   - testSslHandshakeError 

Also manually verified the change by running a 4 node kubernetes cluster 
with 1 JobManagers and 3 TaskManagers, using wildcard DNS certificates and 
executing a stateful streaming program with parallelism set to 2 and verifying 
that all nodes are able to communicate to each other successfully. 

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency):  no
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
  - The serializers: no
  - The runtime per-record code paths (performance sensitive): no
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
  - The S3 file system connector: no

## Documentation

  - Does this pull request introduce a new feature? no
  - If yes, how is the feature documented? not applicable


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/EAlexRojas/flink release-1.4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5789.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5789


commit 202672da7901fe7df912e6a057d6d0c29ccaf0fd
Author: EAlexRojas 
Date:   2018-03-29T14:01:24Z

Using CanonicalHostName instead of IP for SSL coonection on NettyClient




---

[jira] [Commented] (FLINK-9098) ClassLoaderITCase unstable

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419212#comment-16419212
 ] 

Till Rohrmann commented on FLINK-9098:
--

Unblocking 1.5.0 because the {{ClassLoaderITCase}} fails while using the legacy 
code.

> ClassLoaderITCase unstable
> --
>
> Key: FLINK-9098
> URL: https://issues.apache.org/jira/browse/FLINK-9098
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Reporter: Stephan Ewen
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The some savepoint disposal seems to fail, after that all successive tests 
> fail because there are not anymore enough slots.
> Full log: https://api.travis-ci.org/v3/job/356900367/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-9098) ClassLoaderITCase unstable

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-9098:
-
Priority: Critical  (was: Blocker)

> ClassLoaderITCase unstable
> --
>
> Key: FLINK-9098
> URL: https://issues.apache.org/jira/browse/FLINK-9098
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Reporter: Stephan Ewen
>Priority: Critical
> Fix For: 1.5.0
>
>
> The some savepoint disposal seems to fail, after that all successive tests 
> fail because there are not anymore enough slots.
> Full log: https://api.travis-ci.org/v3/job/356900367/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (FLINK-9104) Re-generate REST API documentation for FLIP-6

[Rong Rong (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419211#comment-16419211
 ] 

Rong Rong edited comment on FLINK-9104 at 3/29/18 3:37 PM:
---

Seems like several handlers are missing. I will try going down the path 
[~till.rohrmann] suggested and track down all newly added APIs using derived 
classes from {{MessageHeaders}}.


was (Author: walterddr):
Seems like several handlers are missing. I will try going down the path 
[~till.rohrmann] suggested and track down all newly added APIs from 
{{MessageHeaders}}.

> Re-generate REST API documentation for FLIP-6 
> --
>
> Key: FLINK-9104
> URL: https://issues.apache.org/jira/browse/FLINK-9104
> Project: Flink
>  Issue Type: Bug
>  Components: REST
>Affects Versions: 1.5.0
>Reporter: Gary Yao
>Assignee: Rong Rong
>Priority: Blocker
>  Labels: flip-6
>
> The API documentation is missing for several handlers, e.g., 
> {{SavepointHandlers}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9104) Re-generate REST API documentation for FLIP-6

[Rong Rong (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419211#comment-16419211
 ] 

Rong Rong commented on FLINK-9104:
--

Seems like several handlers are missing. I will try going down the path 
[~till.rohrmann] suggested and track down all newly added APIs from 
{{MessageHeaders}}.

> Re-generate REST API documentation for FLIP-6 
> --
>
> Key: FLINK-9104
> URL: https://issues.apache.org/jira/browse/FLINK-9104
> Project: Flink
>  Issue Type: Bug
>  Components: REST
>Affects Versions: 1.5.0
>Reporter: Gary Yao
>Assignee: Rong Rong
>Priority: Blocker
>  Labels: flip-6
>
> The API documentation is missing for several handlers, e.g., 
> {{SavepointHandlers}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-9104) Re-generate REST API documentation for FLIP-6

[Rong Rong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rong Rong reassigned FLINK-9104:


Assignee: Rong Rong

> Re-generate REST API documentation for FLIP-6 
> --
>
> Key: FLINK-9104
> URL: https://issues.apache.org/jira/browse/FLINK-9104
> Project: Flink
>  Issue Type: Bug
>  Components: REST
>Affects Versions: 1.5.0
>Reporter: Gary Yao
>Assignee: Rong Rong
>Priority: Blocker
>  Labels: flip-6
>
> The API documentation is missing for several handlers, e.g., 
> {{SavepointHandlers}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8986) End-to-end test: REST

[Rong Rong (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419207#comment-16419207
 ] 

Rong Rong commented on FLINK-8986:
--

Thanks [~till.rohrmann] for the info. Yup I can upgrade the doc first then add 
e2e test over all REST calls. Much appreciate the pointers.

> End-to-end test: REST
> -
>
> Key: FLINK-8986
> URL: https://issues.apache.org/jira/browse/FLINK-8986
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Tests
>Reporter: Till Rohrmann
>Assignee: Rong Rong
>Priority: Major
>
> We should add an end-to-end test which verifies that we can use the REST 
> interface to obtain information about a running job.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8985) End-to-end test: CLI

[Rong Rong (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419202#comment-16419202
 ] 

Rong Rong commented on FLINK-8985:
--

Thanks [~till.rohrmann] for the confirmation. Yes, I saw you already have the 
PR up for FLINK-9109. Will start working and incorporate changes accordingly.

> End-to-end test: CLI
> 
>
> Key: FLINK-8985
> URL: https://issues.apache.org/jira/browse/FLINK-8985
> Project: Flink
>  Issue Type: Sub-task
>  Components: Client, Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Rong Rong
>Priority: Major
>
> We should an end-to-end test which verifies that all client commands are 
> working correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8820) FlinkKafkaConsumer010 reads too many bytes

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-8820:
-
Priority: Critical  (was: Blocker)

> FlinkKafkaConsumer010 reads too many bytes
> --
>
> Key: FLINK-8820
> URL: https://issues.apache.org/jira/browse/FLINK-8820
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>Priority: Critical
> Fix For: 1.5.0
>
>
> A user reported that the FlinkKafkaConsumer010 very rarely consumes too many 
> bytes, i.e., the returned message is too large. The application is running 
> for about a year and the problem started to occur after upgrading to Flink 
> 1.4.0.
> The user made a good effort in debugging the problem but was not able to 
> reproduce it in a controlled environment. It seems that the data is correctly 
> stored in Kafka.
> Here's the thread on the thread on the user mailing list for a detailed 
> description of the problem and analysis so far: 
> https://lists.apache.org/thread.html/1d62f616d275e9e23a5215ddf7f5466051be7ea96897d827232fcb4e@%3Cuser.flink.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8731) TwoInputStreamTaskTest flaky on travis

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-8731:
-
Priority: Critical  (was: Blocker)

> TwoInputStreamTaskTest flaky on travis
> --
>
> Key: FLINK-8731
> URL: https://issues.apache.org/jira/browse/FLINK-8731
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming, Tests
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.5.0
>
>
> https://travis-ci.org/zentol/flink/builds/344307861
> {code}
> Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.479 sec <<< 
> FAILURE! - in org.apache.flink.streaming.runtime.tasks.TwoInputStreamTaskTest
> testOpenCloseAndTimestamps(org.apache.flink.streaming.runtime.tasks.TwoInputStreamTaskTest)
>   Time elapsed: 0.05 sec  <<< ERROR!
> java.lang.Exception: error in task
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskTestHarness.waitForTaskCompletion(StreamTaskTestHarness.java:250)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskTestHarness.waitForTaskCompletion(StreamTaskTestHarness.java:233)
>   at 
> org.apache.flink.streaming.runtime.tasks.TwoInputStreamTaskTest.testOpenCloseAndTimestamps(TwoInputStreamTaskTest.java:99)
> Caused by: org.mockito.exceptions.misusing.WrongTypeOfReturnValue: 
> Boolean cannot be returned by getChannelIndex()
> getChannelIndex() should return int
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub 
> spies - 
>- with doReturn|Throw() family of methods. More in javadocs for 
> Mockito.spy() method.
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.UnionInputGate.waitAndGetNextInputGate(UnionInputGate.java:212)
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.UnionInputGate.getNextBufferOrEvent(UnionInputGate.java:158)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:164)
>   at 
> org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:292)
>   at 
> org.apache.flink.streaming.runtime.tasks.TwoInputStreamTask.run(TwoInputStreamTask.java:115)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:308)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskTestHarness$TaskThread.run(StreamTaskTestHarness.java:437)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8731) TwoInputStreamTaskTest flaky on travis

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419195#comment-16419195
 ] 

Till Rohrmann commented on FLINK-8731:
--

Unblocking 1.5.0 since it seems to be a test setup problem with {{Mockito}}.

> TwoInputStreamTaskTest flaky on travis
> --
>
> Key: FLINK-8731
> URL: https://issues.apache.org/jira/browse/FLINK-8731
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming, Tests
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Priority: Blocker
>  Labels: test-stability
> Fix For: 1.5.0
>
>
> https://travis-ci.org/zentol/flink/builds/344307861
> {code}
> Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.479 sec <<< 
> FAILURE! - in org.apache.flink.streaming.runtime.tasks.TwoInputStreamTaskTest
> testOpenCloseAndTimestamps(org.apache.flink.streaming.runtime.tasks.TwoInputStreamTaskTest)
>   Time elapsed: 0.05 sec  <<< ERROR!
> java.lang.Exception: error in task
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskTestHarness.waitForTaskCompletion(StreamTaskTestHarness.java:250)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskTestHarness.waitForTaskCompletion(StreamTaskTestHarness.java:233)
>   at 
> org.apache.flink.streaming.runtime.tasks.TwoInputStreamTaskTest.testOpenCloseAndTimestamps(TwoInputStreamTaskTest.java:99)
> Caused by: org.mockito.exceptions.misusing.WrongTypeOfReturnValue: 
> Boolean cannot be returned by getChannelIndex()
> getChannelIndex() should return int
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub 
> spies - 
>- with doReturn|Throw() family of methods. More in javadocs for 
> Mockito.spy() method.
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.UnionInputGate.waitAndGetNextInputGate(UnionInputGate.java:212)
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.UnionInputGate.getNextBufferOrEvent(UnionInputGate.java:158)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:164)
>   at 
> org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:292)
>   at 
> org.apache.flink.streaming.runtime.tasks.TwoInputStreamTask.run(TwoInputStreamTask.java:115)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:308)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskTestHarness$TaskThread.run(StreamTaskTestHarness.java:437)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8820) FlinkKafkaConsumer010 reads too many bytes

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419193#comment-16419193
 ] 

Till Rohrmann commented on FLINK-8820:
--

I would like to unblock 1.5.0 from this issue since it seems to exist since 
Flink 1.4.0. Moreover, we are currently still investigating. We should 
definitely keep an eye on this issue.

> FlinkKafkaConsumer010 reads too many bytes
> --
>
> Key: FLINK-8820
> URL: https://issues.apache.org/jira/browse/FLINK-8820
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>Priority: Blocker
> Fix For: 1.5.0
>
>
> A user reported that the FlinkKafkaConsumer010 very rarely consumes too many 
> bytes, i.e., the returned message is too large. The application is running 
> for about a year and the problem started to occur after upgrading to Flink 
> 1.4.0.
> The user made a good effort in debugging the problem but was not able to 
> reproduce it in a controlled environment. It seems that the data is correctly 
> stored in Kafka.
> Here's the thread on the thread on the user mailing list for a detailed 
> description of the problem and analysis so far: 
> https://lists.apache.org/thread.html/1d62f616d275e9e23a5215ddf7f5466051be7ea96897d827232fcb4e@%3Cuser.flink.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8891) RestServerEndpoint can bind on local address only

[Gary Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao closed FLINK-8891.
---
Resolution: Fixed

> RestServerEndpoint can bind on local address only
> -
>
> Key: FLINK-8891
> URL: https://issues.apache.org/jira/browse/FLINK-8891
> Project: Flink
>  Issue Type: Bug
>  Components: REST, YARN
>Affects Versions: 1.5.0
> Environment: EC2 AMI debian-jessie-amd64-hvm-2017-01-15-1221-ebs 
> (ami-5900cc36)
> Hadoop 2.8.3
> Flink commit 80020cb5866c8bac67a48f89aa481de7de262f83
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Blocker
>  Labels: flip-6
>
> *Description*
> When deploying a Flink session on YARN, the {{DispatcherRestEndpoint}} may 
> incorrectly bind on a local address. When this happens, the job submission 
> and all REST API calls using a non-local address will fail. Setting 
> {{rest.address: 0.0.0.0}} in {{flink-conf.yaml}} has no effect because the 
> value is overridden.
> *znode leader contents*
> {noformat}
> [zk: localhost:2181(CONNECTED) 3] get 
> /flink/application_1520439896153_0001/leader/rest_server_lock
> ??whttp://127.0.1.1:56299srjava.util.UUIDm?/J
>  leastSigBitsJ
>   
> mostSigBitsxp??L???g?M??KFK
> cZxid = 0x1000a
> ctime = Wed Mar 07 16:25:21 UTC 2018
> mZxid = 0x1000a
> mtime = Wed Mar 07 16:25:21 UTC 2018
> pZxid = 0x1000a
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x5620147c122
> dataLength = 106
> numChildren = 0
> {noformat}
> *Contents of {{/etc/hosts}}*
> {noformat}
> 127.0.1.1 ip-172-31-36-187.eu-central-1.compute.internal ip-172-31-36-187
> 127.0.0.1 localhost
> # The following lines are desirable for IPv6 capable hosts
> ::1 ip6-localhost ip6-loopback
> fe00::0 ip6-localnet
> ff00::0 ip6-mcastprefix
> ff02::1 ip6-allnodes
> ff02::2 ip6-allrouters
> ff02::3 ip6-allhosts
> {noformat}
> Note that without the first line, the problem does not appear.
> *Error message & Stacktrace*
> {noformat}
> 2018-03-07 16:25:24,267 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor   - Found 
> application JobManager host name 
> 'ip-172-31-44-106.eu-central-1.compute.internal' and port '56299' from 
> supplied application id 'application_1520439896153_0001'
> Using the parallelism provided by the remote cluster (0). To use another 
> parallelism, set it at the ./bin/flink client.
> Starting execution of program
> STDERR:
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: Could not submit 
> job 6243b830a6cb1a0b6605a15a7d3d81d4.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:231)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:457)
>   at 
> org.apache.flink.client.program.DetachedEnvironment.finalizeExecute(DetachedEnvironment.java:77)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:403)
>   at 
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:780)
>   at 
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:274)
>   at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:209)
>   at 
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1019)
>   at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1095)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>   at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1095)
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to 
> submit JobGraph.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$4(RestClusterClient.java:327)
>   at 
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
>   at 
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
>   at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>   at 
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
>   at 
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:196)
>   at 
> 

[jira] [Reopened] (FLINK-8891) RestServerEndpoint can bind on local address only

[Gary Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao reopened FLINK-8891:
-

> RestServerEndpoint can bind on local address only
> -
>
> Key: FLINK-8891
> URL: https://issues.apache.org/jira/browse/FLINK-8891
> Project: Flink
>  Issue Type: Bug
>  Components: REST, YARN
>Affects Versions: 1.5.0
> Environment: EC2 AMI debian-jessie-amd64-hvm-2017-01-15-1221-ebs 
> (ami-5900cc36)
> Hadoop 2.8.3
> Flink commit 80020cb5866c8bac67a48f89aa481de7de262f83
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Blocker
>  Labels: flip-6
>
> *Description*
> When deploying a Flink session on YARN, the {{DispatcherRestEndpoint}} may 
> incorrectly bind on a local address. When this happens, the job submission 
> and all REST API calls using a non-local address will fail. Setting 
> {{rest.address: 0.0.0.0}} in {{flink-conf.yaml}} has no effect because the 
> value is overridden.
> *znode leader contents*
> {noformat}
> [zk: localhost:2181(CONNECTED) 3] get 
> /flink/application_1520439896153_0001/leader/rest_server_lock
> ??whttp://127.0.1.1:56299srjava.util.UUIDm?/J
>  leastSigBitsJ
>   
> mostSigBitsxp??L???g?M??KFK
> cZxid = 0x1000a
> ctime = Wed Mar 07 16:25:21 UTC 2018
> mZxid = 0x1000a
> mtime = Wed Mar 07 16:25:21 UTC 2018
> pZxid = 0x1000a
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x5620147c122
> dataLength = 106
> numChildren = 0
> {noformat}
> *Contents of {{/etc/hosts}}*
> {noformat}
> 127.0.1.1 ip-172-31-36-187.eu-central-1.compute.internal ip-172-31-36-187
> 127.0.0.1 localhost
> # The following lines are desirable for IPv6 capable hosts
> ::1 ip6-localhost ip6-loopback
> fe00::0 ip6-localnet
> ff00::0 ip6-mcastprefix
> ff02::1 ip6-allnodes
> ff02::2 ip6-allrouters
> ff02::3 ip6-allhosts
> {noformat}
> Note that without the first line, the problem does not appear.
> *Error message & Stacktrace*
> {noformat}
> 2018-03-07 16:25:24,267 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor   - Found 
> application JobManager host name 
> 'ip-172-31-44-106.eu-central-1.compute.internal' and port '56299' from 
> supplied application id 'application_1520439896153_0001'
> Using the parallelism provided by the remote cluster (0). To use another 
> parallelism, set it at the ./bin/flink client.
> Starting execution of program
> STDERR:
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: Could not submit 
> job 6243b830a6cb1a0b6605a15a7d3d81d4.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:231)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:457)
>   at 
> org.apache.flink.client.program.DetachedEnvironment.finalizeExecute(DetachedEnvironment.java:77)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:403)
>   at 
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:780)
>   at 
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:274)
>   at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:209)
>   at 
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1019)
>   at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1095)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>   at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1095)
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to 
> submit JobGraph.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$4(RestClusterClient.java:327)
>   at 
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
>   at 
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
>   at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>   at 
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
>   at 
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:196)
>   at 
> 

[jira] [Updated] (FLINK-8891) RestServerEndpoint can bind on local address only

[Gary Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao updated FLINK-8891:

Fix Version/s: (was: 1.5.0)

> RestServerEndpoint can bind on local address only
> -
>
> Key: FLINK-8891
> URL: https://issues.apache.org/jira/browse/FLINK-8891
> Project: Flink
>  Issue Type: Bug
>  Components: REST, YARN
>Affects Versions: 1.5.0
> Environment: EC2 AMI debian-jessie-amd64-hvm-2017-01-15-1221-ebs 
> (ami-5900cc36)
> Hadoop 2.8.3
> Flink commit 80020cb5866c8bac67a48f89aa481de7de262f83
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Blocker
>  Labels: flip-6
>
> *Description*
> When deploying a Flink session on YARN, the {{DispatcherRestEndpoint}} may 
> incorrectly bind on a local address. When this happens, the job submission 
> and all REST API calls using a non-local address will fail. Setting 
> {{rest.address: 0.0.0.0}} in {{flink-conf.yaml}} has no effect because the 
> value is overridden.
> *znode leader contents*
> {noformat}
> [zk: localhost:2181(CONNECTED) 3] get 
> /flink/application_1520439896153_0001/leader/rest_server_lock
> ??whttp://127.0.1.1:56299srjava.util.UUIDm?/J
>  leastSigBitsJ
>   
> mostSigBitsxp??L???g?M??KFK
> cZxid = 0x1000a
> ctime = Wed Mar 07 16:25:21 UTC 2018
> mZxid = 0x1000a
> mtime = Wed Mar 07 16:25:21 UTC 2018
> pZxid = 0x1000a
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x5620147c122
> dataLength = 106
> numChildren = 0
> {noformat}
> *Contents of {{/etc/hosts}}*
> {noformat}
> 127.0.1.1 ip-172-31-36-187.eu-central-1.compute.internal ip-172-31-36-187
> 127.0.0.1 localhost
> # The following lines are desirable for IPv6 capable hosts
> ::1 ip6-localhost ip6-loopback
> fe00::0 ip6-localnet
> ff00::0 ip6-mcastprefix
> ff02::1 ip6-allnodes
> ff02::2 ip6-allrouters
> ff02::3 ip6-allhosts
> {noformat}
> Note that without the first line, the problem does not appear.
> *Error message & Stacktrace*
> {noformat}
> 2018-03-07 16:25:24,267 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor   - Found 
> application JobManager host name 
> 'ip-172-31-44-106.eu-central-1.compute.internal' and port '56299' from 
> supplied application id 'application_1520439896153_0001'
> Using the parallelism provided by the remote cluster (0). To use another 
> parallelism, set it at the ./bin/flink client.
> Starting execution of program
> STDERR:
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: Could not submit 
> job 6243b830a6cb1a0b6605a15a7d3d81d4.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:231)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:457)
>   at 
> org.apache.flink.client.program.DetachedEnvironment.finalizeExecute(DetachedEnvironment.java:77)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:403)
>   at 
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:780)
>   at 
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:274)
>   at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:209)
>   at 
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1019)
>   at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1095)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>   at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1095)
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to 
> submit JobGraph.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$4(RestClusterClient.java:327)
>   at 
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
>   at 
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
>   at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>   at 
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
>   at 
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:196)
>   at 
> 

[jira] [Commented] (FLINK-8897) Rowtime materialization causes "mismatched type" AssertionError

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419189#comment-16419189
 ] 

Till Rohrmann commented on FLINK-8897:
--

[~twalthr] is this a problem introduced with Flink 1.5.0 or does it exist for 
longer?

> Rowtime materialization causes "mismatched type" AssertionError
> ---
>
> Key: FLINK-8897
> URL: https://issues.apache.org/jira/browse/FLINK-8897
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Reporter: Xingcan Cui
>Assignee: Timo Walther
>Priority: Blocker
> Fix For: 1.5.0
>
>
> As raised in [this 
> thread|https://lists.apache.org/thread.html/e2ea38aa7ae224d7481145334955d84243690e9aad10d58310bdb8e7@%3Cuser.flink.apache.org%3E],
>  the query created by the following code will throw a calcite "mismatch type" 
> ({{Timestamp(3)}} and {{TimeIndicator}}) exception.
> {code:java}
> String sql1 = "select id, eventTs as t1, count(*) over (partition by id order 
> by eventTs rows between 100 preceding and current row) as cnt1 from myTable1";
> String sql2 = "select distinct id as r_id, eventTs as t2, count(*) over 
> (partition by id order by eventTs rows between 50 preceding and current row) 
> as cnt2 from myTable2";
> Table left = tableEnv.sqlQuery(sql1);
> Table right = tableEnv.sqlQuery(sql2);
> left.join(right).where("id === r_id && t1 === t2").select("id, 
> t1").writeToSink(...)
> {code}
> The logical plan is as follows.
> {code}
> LogicalProject(id=[$0], t1=[$1])
>   LogicalFilter(condition=[AND(=($0, $3), =($1, $4))])
> LogicalJoin(condition=[true], joinType=[inner])
>   LogicalAggregate(group=[{0, 1, 2}])
> LogicalWindow(window#0=[window(partition {0} order by [1] rows 
> between $2 PRECEDING and CURRENT ROW aggs [COUNT()])])
>   LogicalProject(id=[$0], eventTs=[$3])
> LogicalTableScan(table=[[_DataStreamTable_0]])
>   LogicalAggregate(group=[{0, 1, 2}])
> LogicalWindow(window#0=[window(partition {0} order by [1] rows 
> between $2 PRECEDING and CURRENT ROW aggs [COUNT()])])
>   LogicalProject(id=[$0], eventTs=[$3])
> LogicalTableScan(table=[[_DataStreamTable_0]])
> {code}
> That is because the the rowtime field after an aggregation will be 
> materialized while the {{RexInputRef}} type for the filter's operands ({{t1 
> === t2}}) is still {{TimeIndicator}}. We should make them unified.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8891) RestServerEndpoint can bind on local address only

[Gary Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao closed FLINK-8891.
---
Resolution: Fixed

Fixed together with FLINK-8843

> RestServerEndpoint can bind on local address only
> -
>
> Key: FLINK-8891
> URL: https://issues.apache.org/jira/browse/FLINK-8891
> Project: Flink
>  Issue Type: Bug
>  Components: REST, YARN
>Affects Versions: 1.5.0
> Environment: EC2 AMI debian-jessie-amd64-hvm-2017-01-15-1221-ebs 
> (ami-5900cc36)
> Hadoop 2.8.3
> Flink commit 80020cb5866c8bac67a48f89aa481de7de262f83
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> *Description*
> When deploying a Flink session on YARN, the {{DispatcherRestEndpoint}} may 
> incorrectly bind on a local address. When this happens, the job submission 
> and all REST API calls using a non-local address will fail. Setting 
> {{rest.address: 0.0.0.0}} in {{flink-conf.yaml}} has no effect because the 
> value is overridden.
> *znode leader contents*
> {noformat}
> [zk: localhost:2181(CONNECTED) 3] get 
> /flink/application_1520439896153_0001/leader/rest_server_lock
> ??whttp://127.0.1.1:56299srjava.util.UUIDm?/J
>  leastSigBitsJ
>   
> mostSigBitsxp??L???g?M??KFK
> cZxid = 0x1000a
> ctime = Wed Mar 07 16:25:21 UTC 2018
> mZxid = 0x1000a
> mtime = Wed Mar 07 16:25:21 UTC 2018
> pZxid = 0x1000a
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x5620147c122
> dataLength = 106
> numChildren = 0
> {noformat}
> *Contents of {{/etc/hosts}}*
> {noformat}
> 127.0.1.1 ip-172-31-36-187.eu-central-1.compute.internal ip-172-31-36-187
> 127.0.0.1 localhost
> # The following lines are desirable for IPv6 capable hosts
> ::1 ip6-localhost ip6-loopback
> fe00::0 ip6-localnet
> ff00::0 ip6-mcastprefix
> ff02::1 ip6-allnodes
> ff02::2 ip6-allrouters
> ff02::3 ip6-allhosts
> {noformat}
> Note that without the first line, the problem does not appear.
> *Error message & Stacktrace*
> {noformat}
> 2018-03-07 16:25:24,267 INFO  
> org.apache.flink.yarn.AbstractYarnClusterDescriptor   - Found 
> application JobManager host name 
> 'ip-172-31-44-106.eu-central-1.compute.internal' and port '56299' from 
> supplied application id 'application_1520439896153_0001'
> Using the parallelism provided by the remote cluster (0). To use another 
> parallelism, set it at the ./bin/flink client.
> Starting execution of program
> STDERR:
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: Could not submit 
> job 6243b830a6cb1a0b6605a15a7d3d81d4.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:231)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:457)
>   at 
> org.apache.flink.client.program.DetachedEnvironment.finalizeExecute(DetachedEnvironment.java:77)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:403)
>   at 
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:780)
>   at 
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:274)
>   at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:209)
>   at 
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1019)
>   at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1095)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>   at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1095)
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to 
> submit JobGraph.
>   at 
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$4(RestClusterClient.java:327)
>   at 
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
>   at 
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
>   at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>   at 
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
>   at 
> 

[jira] [Commented] (FLINK-7219) Current allocate strategy cann‘t achieve the optimal effect with input's location

[Sihua Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419188#comment-16419188
 ] 

Sihua Zhou commented on FLINK-7219:
---

[~aljoscha] Got it and sorry...

> Current allocate strategy cann‘t achieve the optimal effect with input's 
> location
> -
>
> Key: FLINK-7219
> URL: https://issues.apache.org/jira/browse/FLINK-7219
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.3.1
>Reporter: Sihua Zhou
>Priority: Major
>
> This is second subtask of issue 
> [FLINK-7153|https://issues.apache.org/jira/browse/FLINK-7153?filter=-2].
> Current allocate strategy can't allocate the slot optimize.  Here is the test 
> case:
> {code}
> JobVertex v1 = new JobVertex("v1", jid1);
> JobVertex v2 = new JobVertex("v2", jid2);
> SlotSharingGroup group = new SlotSharingGroup();
> v1.setSlotSharingGroup(group);
> v2.setSlotSharingGroup(group);
> v1.setParallelism(2);
> v2.setParallelism(4);
> v1.setInvokableClass(BatchTask.class);
> v2.setInvokableClass(BatchTask.class);
> v2.connectNewDataSetAsInput(v1, DistributionPattern.POINTWISE, 
> ResultPartitionType.PIPELINED_BOUNDED);
> {code}
> Currently, after allocate for v1,v2, we got a local partition and three 
> remote partition. But actually, it should be 2 local partition and 2 remote 
> partition. 
> The causes of the above problems is becuase that the current allocate 
> strategy is allocate the resource for execution one by one(if the execution 
> can allocate from SlotGroup than get it, Otherwise ask for a new one for it). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8867) Rocksdb checkpointing failing with fs.default-scheme: hdfs:// config

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-8867:
-
Priority: Critical  (was: Blocker)

> Rocksdb checkpointing failing with fs.default-scheme: hdfs:// config
> 
>
> Key: FLINK-8867
> URL: https://issues.apache.org/jira/browse/FLINK-8867
> Project: Flink
>  Issue Type: Bug
>  Components: Configuration, State Backends, Checkpointing, YARN
>Affects Versions: 1.4.1, 1.4.2
>Reporter: Shashank Agarwal
>Assignee: Stefan Richter
>Priority: Critical
> Fix For: 1.5.0
>
>
> In our setup, when we put an entry in our Flink_conf file for default schema.
> {code}
> fs.default-scheme: hdfs://mydomain.com:8020/flink
> {code}
> Than application with rocksdb state backend fails with the following 
> exception. When we remove this config it works fine. It's working fine with 
> other state backends.
> {code}
> AsynchronousException{java.lang.Exception: Could not materialize checkpoint 1 
> for operator order ip stream (1/1).}
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:948)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.Exception: Could not materialize checkpoint 1 for 
> operator order ip stream (1/1).
>   ... 6 more
> Caused by: java.util.concurrent.ExecutionException: 
> java.lang.IllegalStateException
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>   at 
> org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:894)
>   ... 5 more
>   Suppressed: java.lang.Exception: Could not properly cancel managed 
> keyed state future.
>   at 
> org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:91)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.cleanup(StreamTask.java:976)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:939)
>   ... 5 more
>   Caused by: java.util.concurrent.ExecutionException: 
> java.lang.IllegalStateException
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>   at 
> org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43)
>   at 
> org.apache.flink.runtime.state.StateUtil.discardStateFuture(StateUtil.java:66)
>   at 
> org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:89)
>   ... 7 more
>   Caused by: java.lang.IllegalStateException
>   at 
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:179)
>   at 
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalSnapshotOperation.materializeSnapshot(RocksDBKeyedStateBackend.java:926)
>   at 
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$1.call(RocksDBKeyedStateBackend.java:389)
>   at 
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$1.call(RocksDBKeyedStateBackend.java:386)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:40)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:894)
>   ... 5 more
>   [CIRCULAR REFERENCE:java.lang.IllegalStateException]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8867) Rocksdb checkpointing failing with fs.default-scheme: hdfs:// config

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419182#comment-16419182
 ] 

Till Rohrmann commented on FLINK-8867:
--

Unblocking 1.5.0 from this issue since it seems to exist for longer. Would 
still like to get a fix for it in 1.5.0 if possible.

> Rocksdb checkpointing failing with fs.default-scheme: hdfs:// config
> 
>
> Key: FLINK-8867
> URL: https://issues.apache.org/jira/browse/FLINK-8867
> Project: Flink
>  Issue Type: Bug
>  Components: Configuration, State Backends, Checkpointing, YARN
>Affects Versions: 1.4.1, 1.4.2
>Reporter: Shashank Agarwal
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.5.0
>
>
> In our setup, when we put an entry in our Flink_conf file for default schema.
> {code}
> fs.default-scheme: hdfs://mydomain.com:8020/flink
> {code}
> Than application with rocksdb state backend fails with the following 
> exception. When we remove this config it works fine. It's working fine with 
> other state backends.
> {code}
> AsynchronousException{java.lang.Exception: Could not materialize checkpoint 1 
> for operator order ip stream (1/1).}
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:948)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.Exception: Could not materialize checkpoint 1 for 
> operator order ip stream (1/1).
>   ... 6 more
> Caused by: java.util.concurrent.ExecutionException: 
> java.lang.IllegalStateException
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>   at 
> org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:894)
>   ... 5 more
>   Suppressed: java.lang.Exception: Could not properly cancel managed 
> keyed state future.
>   at 
> org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:91)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.cleanup(StreamTask.java:976)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:939)
>   ... 5 more
>   Caused by: java.util.concurrent.ExecutionException: 
> java.lang.IllegalStateException
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>   at 
> org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43)
>   at 
> org.apache.flink.runtime.state.StateUtil.discardStateFuture(StateUtil.java:66)
>   at 
> org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:89)
>   ... 7 more
>   Caused by: java.lang.IllegalStateException
>   at 
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:179)
>   at 
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalSnapshotOperation.materializeSnapshot(RocksDBKeyedStateBackend.java:926)
>   at 
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$1.call(RocksDBKeyedStateBackend.java:389)
>   at 
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$1.call(RocksDBKeyedStateBackend.java:386)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:40)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:894)
>   ... 5 more
>   [CIRCULAR REFERENCE:java.lang.IllegalStateException]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8837) Move DataStreamUtils to package 'experimental'.

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419171#comment-16419171
 ] 

Till Rohrmann commented on FLINK-8837:
--

+1 for the {{@Experimental}} annotation.

Also +1 for [~phoenixjiangnan] to take a stab at it from my side.

> Move DataStreamUtils to package 'experimental'.
> ---
>
> Key: FLINK-8837
> URL: https://issues.apache.org/jira/browse/FLINK-8837
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Reporter: Stephan Ewen
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The class {{DataStreamUtils}} came from 'flink-contrib' and now accidentally 
> moved to the fully supported API packages. It should be in package 
> 'experimental' to properly communicate that it is not guaranteed to be API 
> stable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8825) Disallow new String() without charset in checkstyle

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-8825:
-
Priority: Critical  (was: Blocker)

> Disallow new String() without charset in checkstyle
> ---
>
> Key: FLINK-8825
> URL: https://issues.apache.org/jira/browse/FLINK-8825
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Reporter: Aljoscha Krettek
>Priority: Critical
> Fix For: 1.5.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8825) Disallow new String() without charset in checkstyle

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419164#comment-16419164
 ] 

Till Rohrmann commented on FLINK-8825:
--

Not a blocker

> Disallow new String() without charset in checkstyle
> ---
>
> Key: FLINK-8825
> URL: https://issues.apache.org/jira/browse/FLINK-8825
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Reporter: Aljoscha Krettek
>Priority: Critical
> Fix For: 1.5.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8825) Disallow new String() without charset in checkstyle

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-8825:
-
Priority: Major  (was: Critical)

> Disallow new String() without charset in checkstyle
> ---
>
> Key: FLINK-8825
> URL: https://issues.apache.org/jira/browse/FLINK-8825
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Reporter: Aljoscha Krettek
>Priority: Major
> Fix For: 1.5.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8512) HAQueryableStateFsBackendITCase unstable on Travis

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated FLINK-8512:

Fix Version/s: (was: 1.4.3)
   (was: 1.5.0)

> HAQueryableStateFsBackendITCase unstable on Travis
> --
>
> Key: FLINK-8512
> URL: https://issues.apache.org/jira/browse/FLINK-8512
> Project: Flink
>  Issue Type: Bug
>  Components: Queryable State, Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Priority: Blocker
>  Labels: test-stability
>
> {{HAQueryableStateFsBackendITCase}} is unstable on Travis.
> In the logs one can see that there is an {{AssertionError}} in the 
> {{globalEventExecutor-1-1}} {{Thread}}. This indicates that assertions are 
> not properly propagated and simply swallowed.
>  
> https://travis-ci.org/apache/flink/jobs/333250401



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8512) HAQueryableStateFsBackendITCase unstable on Travis

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-8512.
---
Resolution: Not A Bug

> HAQueryableStateFsBackendITCase unstable on Travis
> --
>
> Key: FLINK-8512
> URL: https://issues.apache.org/jira/browse/FLINK-8512
> Project: Flink
>  Issue Type: Bug
>  Components: Queryable State, Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Priority: Blocker
>  Labels: test-stability
>
> {{HAQueryableStateFsBackendITCase}} is unstable on Travis.
> In the logs one can see that there is an {{AssertionError}} in the 
> {{globalEventExecutor-1-1}} {{Thread}}. This indicates that assertions are 
> not properly propagated and simply swallowed.
>  
> https://travis-ci.org/apache/flink/jobs/333250401



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (FLINK-8512) HAQueryableStateFsBackendITCase unstable on Travis

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reopened FLINK-8512:
-

Reopen to remove fixVersion

> HAQueryableStateFsBackendITCase unstable on Travis
> --
>
> Key: FLINK-8512
> URL: https://issues.apache.org/jira/browse/FLINK-8512
> Project: Flink
>  Issue Type: Bug
>  Components: Queryable State, Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Priority: Blocker
>  Labels: test-stability
>
> {{HAQueryableStateFsBackendITCase}} is unstable on Travis.
> In the logs one can see that there is an {{AssertionError}} in the 
> {{globalEventExecutor-1-1}} {{Thread}}. This indicates that assertions are 
> not properly propagated and simply swallowed.
>  
> https://travis-ci.org/apache/flink/jobs/333250401



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-9080) Flink Scheduler goes OOM, suspecting a memory leak

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated FLINK-9080:

Fix Version/s: 1.5.0

> Flink Scheduler goes OOM, suspecting a memory leak
> --
>
> Key: FLINK-9080
> URL: https://issues.apache.org/jira/browse/FLINK-9080
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.4.0
>Reporter: Rohit Singh
>Priority: Blocker
> Fix For: 1.5.0
>
> Attachments: Top Level packages.JPG, Top level classes.JPG, 
> classesloaded vs unloaded.png
>
>
> Running FLink version 1.4.0. on mesos,scheduler running along  with job 
> manager in single container, whereas task managers running in seperate 
> containers.
> Couple of jobs were running continously, Flink scheduler was working 
> properlyalong with task managers. Due to some change in data, one of the jobs 
> started failing continuously. In the meantime,there was a surge in  flink 
> scheduler memory usually eventually died out off OOM
>  
> Memory dump analysis was done, 
> Following were findings  !Top Level packages.JPG!!Top level classes.JPG!
>  *  Majority of top loaded packages retaining heap indicated towards 
> Flinkuserclassloader, glassfish(jersey library), Finalizer classes. (Top 
> level package image)
>  * Top level classes were of Flinkuserclassloader, (Top Level class image)
>  * The number of classes loaded vs unloaded was quite less  PFA,inspite of 
> adding jvm options of -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled , 
> PFAclassloaded vs unloaded graph, scheduler was restarted 3 times
>  * There were custom classes as well which were duplicated during subsequent 
> class uploads
> PFA all the images of heap dump.  Can you suggest some pointers on as to how 
> to overcome this issue.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8803) Mini Cluster Shutdown with HA unstable, causing test failures

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419155#comment-16419155
 ] 

Till Rohrmann commented on FLINK-8803:
--

Unblocking 1.5.0 because the problem affects the legacy code 
{{FlinkMiniCluster}}.

> Mini Cluster Shutdown with HA unstable, causing test failures
> -
>
> Key: FLINK-8803
> URL: https://issues.apache.org/jira/browse/FLINK-8803
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Reporter: Stephan Ewen
>Priority: Blocker
> Fix For: 1.5.0
>
>
> When the {{FlinkMiniCluster}} is created for HA tests with ZooKeeper, the 
> shutdown is unstable.
> It looks like ZooKeeper may be shut down before the JobManager is shut down, 
> causing the shutdown procedure of the JobManager (specifically 
> {{ZooKeeperSubmittedJobGraphStore.removeJobGraph}}) to block until tests time 
> out.
> Full log: https://api.travis-ci.org/v3/job/346853707/log.txt
> Note that no ZK threads are alive any more, seems ZK is shut down already.
> Relevant Stack Traces:
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7f973800a800 nid=0x43b4 waiting on 
> condition [0x7f973eb0b000]
>java.lang.Thread.State: TIMED_WAITING (parking)
>   at sun.misc.Unsafe.park(Native Method)
>   - parking to wait for  <0x8966cf18> (a 
> scala.concurrent.impl.Promise$CompletionLatch)
>   at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
>   at 
> scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:212)
>   at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:222)
>   at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:157)
>   at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:169)
>   at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:169)
>   at 
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>   at scala.concurrent.Await$.ready(package.scala:169)
>   at 
> org.apache.flink.runtime.minicluster.FlinkMiniCluster.startInternalShutdown(FlinkMiniCluster.scala:469)
>   at 
> org.apache.flink.runtime.minicluster.FlinkMiniCluster.stop(FlinkMiniCluster.scala:435)
>   at 
> org.apache.flink.runtime.minicluster.FlinkMiniCluster.closeAsync(FlinkMiniCluster.scala:719)
>   at 
> org.apache.flink.test.util.MiniClusterResource.after(MiniClusterResource.java:104)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
> ...
> {code}
> {code}
> "flink-akka.actor.default-dispatcher-2" #1012 prio=5 os_prio=0 
> tid=0x7f97394fa800 nid=0x3328 waiting on condition [0x7f971db29000]
>java.lang.Thread.State: TIMED_WAITING (parking)
>   at sun.misc.Unsafe.park(Native Method)
>   - parking to wait for  <0x87f82a70> (a 
> java.util.concurrent.CountDownLatch$Sync)
>   at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
>   at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
>   at 
> org.apache.flink.shaded.curator.org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:336)
>   at 
> org.apache.flink.shaded.curator.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
>   at 
> org.apache.flink.shaded.curator.org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:241)
>   at 
> org.apache.flink.shaded.curator.org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:225)
>   at 
> org.apache.flink.shaded.curator.org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:35)
>   at 
> org.apache.flink.runtime.zookeeper.ZooKeeperStateHandleStore.release(ZooKeeperStateHandleStore.java:478)
>   at 
> org.apache.flink.runtime.zookeeper.ZooKeeperStateHandleStore.releaseAndTryRemove(ZooKeeperStateHandleStore.java:435)
>   at 
> org.apache.flink.runtime.zookeeper.ZooKeeperStateHandleStore.releaseAndTryRemove(ZooKeeperStateHandleStore.java:405)
>   at 
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.removeJobGraph(ZooKeeperSubmittedJobGraphStore.java:266)
>   - locked <0x807f4258> (a 

[jira] [Updated] (FLINK-8803) Mini Cluster Shutdown with HA unstable, causing test failures

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-8803:
-
Priority: Critical  (was: Blocker)

> Mini Cluster Shutdown with HA unstable, causing test failures
> -
>
> Key: FLINK-8803
> URL: https://issues.apache.org/jira/browse/FLINK-8803
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Reporter: Stephan Ewen
>Priority: Critical
> Fix For: 1.5.0
>
>
> When the {{FlinkMiniCluster}} is created for HA tests with ZooKeeper, the 
> shutdown is unstable.
> It looks like ZooKeeper may be shut down before the JobManager is shut down, 
> causing the shutdown procedure of the JobManager (specifically 
> {{ZooKeeperSubmittedJobGraphStore.removeJobGraph}}) to block until tests time 
> out.
> Full log: https://api.travis-ci.org/v3/job/346853707/log.txt
> Note that no ZK threads are alive any more, seems ZK is shut down already.
> Relevant Stack Traces:
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7f973800a800 nid=0x43b4 waiting on 
> condition [0x7f973eb0b000]
>java.lang.Thread.State: TIMED_WAITING (parking)
>   at sun.misc.Unsafe.park(Native Method)
>   - parking to wait for  <0x8966cf18> (a 
> scala.concurrent.impl.Promise$CompletionLatch)
>   at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
>   at 
> scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:212)
>   at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:222)
>   at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:157)
>   at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:169)
>   at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:169)
>   at 
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>   at scala.concurrent.Await$.ready(package.scala:169)
>   at 
> org.apache.flink.runtime.minicluster.FlinkMiniCluster.startInternalShutdown(FlinkMiniCluster.scala:469)
>   at 
> org.apache.flink.runtime.minicluster.FlinkMiniCluster.stop(FlinkMiniCluster.scala:435)
>   at 
> org.apache.flink.runtime.minicluster.FlinkMiniCluster.closeAsync(FlinkMiniCluster.scala:719)
>   at 
> org.apache.flink.test.util.MiniClusterResource.after(MiniClusterResource.java:104)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
> ...
> {code}
> {code}
> "flink-akka.actor.default-dispatcher-2" #1012 prio=5 os_prio=0 
> tid=0x7f97394fa800 nid=0x3328 waiting on condition [0x7f971db29000]
>java.lang.Thread.State: TIMED_WAITING (parking)
>   at sun.misc.Unsafe.park(Native Method)
>   - parking to wait for  <0x87f82a70> (a 
> java.util.concurrent.CountDownLatch$Sync)
>   at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
>   at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
>   at 
> org.apache.flink.shaded.curator.org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:336)
>   at 
> org.apache.flink.shaded.curator.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
>   at 
> org.apache.flink.shaded.curator.org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:241)
>   at 
> org.apache.flink.shaded.curator.org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:225)
>   at 
> org.apache.flink.shaded.curator.org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:35)
>   at 
> org.apache.flink.runtime.zookeeper.ZooKeeperStateHandleStore.release(ZooKeeperStateHandleStore.java:478)
>   at 
> org.apache.flink.runtime.zookeeper.ZooKeeperStateHandleStore.releaseAndTryRemove(ZooKeeperStateHandleStore.java:435)
>   at 
> org.apache.flink.runtime.zookeeper.ZooKeeperStateHandleStore.releaseAndTryRemove(ZooKeeperStateHandleStore.java:405)
>   at 
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.removeJobGraph(ZooKeeperSubmittedJobGraphStore.java:266)
>   - locked <0x807f4258> (a java.lang.Object)
>   at 
> 

[jira] [Updated] (FLINK-8803) Mini Cluster Shutdown with HA unstable, causing test failures

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-8803:
-
Description: 
When the {{FlinkMiniCluster}} is created for HA tests with ZooKeeper, the 
shutdown is unstable.

It looks like ZooKeeper may be shut down before the JobManager is shut down, 
causing the shutdown procedure of the JobManager (specifically 
{{ZooKeeperSubmittedJobGraphStore.removeJobGraph}}) to block until tests time 
out.

Full log: https://api.travis-ci.org/v3/job/346853707/log.txt

Note that no ZK threads are alive any more, seems ZK is shut down already.

Relevant Stack Traces:

{code}

"main" #1 prio=5 os_prio=0 tid=0x7f973800a800 nid=0x43b4 waiting on 
condition [0x7f973eb0b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x8966cf18> (a 
scala.concurrent.impl.Promise$CompletionLatch)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at 
scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:212)
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:222)
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:157)
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:169)
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:169)
at 
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.ready(package.scala:169)
at 
org.apache.flink.runtime.minicluster.FlinkMiniCluster.startInternalShutdown(FlinkMiniCluster.scala:469)
at 
org.apache.flink.runtime.minicluster.FlinkMiniCluster.stop(FlinkMiniCluster.scala:435)
at 
org.apache.flink.runtime.minicluster.FlinkMiniCluster.closeAsync(FlinkMiniCluster.scala:719)
at 
org.apache.flink.test.util.MiniClusterResource.after(MiniClusterResource.java:104)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
...
{code}

{code}
"flink-akka.actor.default-dispatcher-2" #1012 prio=5 os_prio=0 
tid=0x7f97394fa800 nid=0x3328 waiting on condition [0x7f971db29000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x87f82a70> (a 
java.util.concurrent.CountDownLatch$Sync)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at 
org.apache.flink.shaded.curator.org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:336)
at 
org.apache.flink.shaded.curator.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at 
org.apache.flink.shaded.curator.org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:241)
at 
org.apache.flink.shaded.curator.org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:225)
at 
org.apache.flink.shaded.curator.org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:35)
at 
org.apache.flink.runtime.zookeeper.ZooKeeperStateHandleStore.release(ZooKeeperStateHandleStore.java:478)
at 
org.apache.flink.runtime.zookeeper.ZooKeeperStateHandleStore.releaseAndTryRemove(ZooKeeperStateHandleStore.java:435)
at 
org.apache.flink.runtime.zookeeper.ZooKeeperStateHandleStore.releaseAndTryRemove(ZooKeeperStateHandleStore.java:405)
at 
org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.removeJobGraph(ZooKeeperSubmittedJobGraphStore.java:266)
- locked <0x807f4258> (a java.lang.Object)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$1.apply$mcV$sp(JobManager.scala:1727)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$1.apply(JobManager.scala:1723)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$1.apply(JobManager.scala:1723)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at 

[jira] [Commented] (FLINK-8795) Scala shell broken for Flip6

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419148#comment-16419148
 ] 

Till Rohrmann commented on FLINK-8795:
--

Unblocking 1.5.0 since there is a workaround.

> Scala shell broken for Flip6
> 
>
> Key: FLINK-8795
> URL: https://issues.apache.org/jira/browse/FLINK-8795
> Project: Flink
>  Issue Type: Bug
>Reporter: kant kodali
>Priority: Major
> Fix For: 1.5.0
>
>
> I am trying to run the simple code below after building everything from 
> Flink's github master branch for various reasons. I get an exception below 
> and I wonder what runs on port 9065? and How to fix this exception?
> I followed the instructions from the Flink master branch so I did the 
> following.
> {code:java}
> git clone https://github.com/apache/flink.git 
> cd flink mvn clean package -DskipTests 
> cd build-target
>  ./bin/start-scala-shell.sh local{code}
> {{And Here is the code I ran}}
> {code:java}
> val dataStream = senv.fromElements(1, 2, 3, 4)
> dataStream.countWindowAll(2).sum(0).print()
> senv.execute("My streaming program"){code}
> {{And I finally get this exception}}
> {code:java}
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to 
> submit JobGraph. at 
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$18(RestClusterClient.java:306)
>  at 
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
>  at 
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
>  at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>  at 
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
>  at 
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$222(RestClient.java:196)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:268)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:284)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>  at java.lang.Thread.run(Thread.java:745) Caused by: 
> java.util.concurrent.CompletionException: java.net.ConnectException: 
> Connection refused: localhost/127.0.0.1:9065 at 
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
>  at 
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
>  at 
> java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943) 
> at 
> java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
>  ... 16 more Caused by: java.net.ConnectException: Connection refused: 
> localhost/127.0.0.1:9065 at sun.nio.ch.SocketChannelImpl.checkConnect(Native 
> Method) at 
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at 
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:281){code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-6645) JobMasterTest failed on Travis

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-6645.
---
Resolution: Cannot Reproduce

> JobMasterTest failed on Travis
> --
>
> Key: FLINK-6645
> URL: https://issues.apache.org/jira/browse/FLINK-6645
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Priority: Blocker
>  Labels: test-stability
>
> {code}
> Failed tests: 
>   JobMasterTest.testHeartbeatTimeoutWithResourceManager:220 
> Wanted but not invoked:
> resourceManagerGateway.disconnectJobManager(
> be75c925204aede002136b15238f88b4,
> 
> );
> -> at 
> org.apache.flink.runtime.jobmaster.JobMasterTest.testHeartbeatTimeoutWithResourceManager(JobMasterTest.java:220)
> However, there were other interactions with this mock:
> resourceManagerGateway.registerJobManager(
> 82d774de-ed78-4670-9623-2fc6638fbbf9,
> 11b045ea-8b2b-4df0-aa02-ef4922dfc632,
> jm,
> "b442340a-7d3d-49d8-b440-1a93a5b43bb6",
> be75c925204aede002136b15238f88b4,
> 100 ms
> );
> -> at 
> org.apache.flink.runtime.jobmaster.JobMaster$ResourceManagerConnection$1.invokeRegistration(JobMaster.java:1051)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8795) Scala shell broken for Flip6

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-8795:
-
Priority: Major  (was: Blocker)

> Scala shell broken for Flip6
> 
>
> Key: FLINK-8795
> URL: https://issues.apache.org/jira/browse/FLINK-8795
> Project: Flink
>  Issue Type: Bug
>Reporter: kant kodali
>Priority: Major
> Fix For: 1.5.0
>
>
> I am trying to run the simple code below after building everything from 
> Flink's github master branch for various reasons. I get an exception below 
> and I wonder what runs on port 9065? and How to fix this exception?
> I followed the instructions from the Flink master branch so I did the 
> following.
> {code:java}
> git clone https://github.com/apache/flink.git 
> cd flink mvn clean package -DskipTests 
> cd build-target
>  ./bin/start-scala-shell.sh local{code}
> {{And Here is the code I ran}}
> {code:java}
> val dataStream = senv.fromElements(1, 2, 3, 4)
> dataStream.countWindowAll(2).sum(0).print()
> senv.execute("My streaming program"){code}
> {{And I finally get this exception}}
> {code:java}
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to 
> submit JobGraph. at 
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$18(RestClusterClient.java:306)
>  at 
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
>  at 
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
>  at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>  at 
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
>  at 
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$222(RestClient.java:196)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:268)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:284)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>  at 
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>  at java.lang.Thread.run(Thread.java:745) Caused by: 
> java.util.concurrent.CompletionException: java.net.ConnectException: 
> Connection refused: localhost/127.0.0.1:9065 at 
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
>  at 
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
>  at 
> java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943) 
> at 
> java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
>  ... 16 more Caused by: java.net.ConnectException: Connection refused: 
> localhost/127.0.0.1:9065 at sun.nio.ch.SocketChannelImpl.checkConnect(Native 
> Method) at 
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at 
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
>  at 
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:281){code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (FLINK-6645) JobMasterTest failed on Travis

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reopened FLINK-6645:
-

Reopen to remove fixVersion.

> JobMasterTest failed on Travis
> --
>
> Key: FLINK-6645
> URL: https://issues.apache.org/jira/browse/FLINK-6645
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Priority: Blocker
>  Labels: test-stability
>
> {code}
> Failed tests: 
>   JobMasterTest.testHeartbeatTimeoutWithResourceManager:220 
> Wanted but not invoked:
> resourceManagerGateway.disconnectJobManager(
> be75c925204aede002136b15238f88b4,
> 
> );
> -> at 
> org.apache.flink.runtime.jobmaster.JobMasterTest.testHeartbeatTimeoutWithResourceManager(JobMasterTest.java:220)
> However, there were other interactions with this mock:
> resourceManagerGateway.registerJobManager(
> 82d774de-ed78-4670-9623-2fc6638fbbf9,
> 11b045ea-8b2b-4df0-aa02-ef4922dfc632,
> jm,
> "b442340a-7d3d-49d8-b440-1a93a5b43bb6",
> be75c925204aede002136b15238f88b4,
> 100 ms
> );
> -> at 
> org.apache.flink.runtime.jobmaster.JobMaster$ResourceManagerConnection$1.invokeRegistration(JobMaster.java:1051)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-6645) JobMasterTest failed on Travis

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated FLINK-6645:

Fix Version/s: (was: 1.5.0)

> JobMasterTest failed on Travis
> --
>
> Key: FLINK-6645
> URL: https://issues.apache.org/jira/browse/FLINK-6645
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Priority: Blocker
>  Labels: test-stability
>
> {code}
> Failed tests: 
>   JobMasterTest.testHeartbeatTimeoutWithResourceManager:220 
> Wanted but not invoked:
> resourceManagerGateway.disconnectJobManager(
> be75c925204aede002136b15238f88b4,
> 
> );
> -> at 
> org.apache.flink.runtime.jobmaster.JobMasterTest.testHeartbeatTimeoutWithResourceManager(JobMasterTest.java:220)
> However, there were other interactions with this mock:
> resourceManagerGateway.registerJobManager(
> 82d774de-ed78-4670-9623-2fc6638fbbf9,
> 11b045ea-8b2b-4df0-aa02-ef4922dfc632,
> jm,
> "b442340a-7d3d-49d8-b440-1a93a5b43bb6",
> be75c925204aede002136b15238f88b4,
> 100 ms
> );
> -> at 
> org.apache.flink.runtime.jobmaster.JobMaster$ResourceManagerConnection$1.invokeRegistration(JobMaster.java:1051)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-7219) Current allocate strategy cann‘t achieve the optimal effect with input's location

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-7219.
---
Resolution: Fixed

> Current allocate strategy cann‘t achieve the optimal effect with input's 
> location
> -
>
> Key: FLINK-7219
> URL: https://issues.apache.org/jira/browse/FLINK-7219
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.3.1
>Reporter: Sihua Zhou
>Priority: Major
>
> This is second subtask of issue 
> [FLINK-7153|https://issues.apache.org/jira/browse/FLINK-7153?filter=-2].
> Current allocate strategy can't allocate the slot optimize.  Here is the test 
> case:
> {code}
> JobVertex v1 = new JobVertex("v1", jid1);
> JobVertex v2 = new JobVertex("v2", jid2);
> SlotSharingGroup group = new SlotSharingGroup();
> v1.setSlotSharingGroup(group);
> v2.setSlotSharingGroup(group);
> v1.setParallelism(2);
> v2.setParallelism(4);
> v1.setInvokableClass(BatchTask.class);
> v2.setInvokableClass(BatchTask.class);
> v2.connectNewDataSetAsInput(v1, DistributionPattern.POINTWISE, 
> ResultPartitionType.PIPELINED_BOUNDED);
> {code}
> Currently, after allocate for v1,v2, we got a local partition and three 
> remote partition. But actually, it should be 2 local partition and 2 remote 
> partition. 
> The causes of the above problems is becuase that the current allocate 
> strategy is allocate the resource for execution one by one(if the execution 
> can allocate from SlotGroup than get it, Otherwise ask for a new one for it). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8793) Hide key containing "secret" in web interface

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-8793:
-
Priority: Critical  (was: Blocker)

> Hide key containing "secret" in web interface
> -
>
> Key: FLINK-8793
> URL: https://issues.apache.org/jira/browse/FLINK-8793
> Project: Flink
>  Issue Type: Bug
>  Components: REST
>Affects Versions: 1.4.1
>Reporter: Etienne CARRIERE
>Priority: Critical
> Fix For: 1.5.0
>
>
> Currently, we going in /jobmanager/config on the web interface, the value of 
> the key containing "password" are replaced by "" 
> When using s3 for checkpoint/savepoint configuration on an infrastructure 
> which is not on AWS (where IAM is not possible), the s3.secret-key is 
> revealed from the interface. 
> I propose the same behaviour as key with "password" for key with "secret" 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8793) Hide key containing "secret" in web interface

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419144#comment-16419144
 ] 

Till Rohrmann commented on FLINK-8793:
--

Thanks for reporting this issue [~etienne.carri...@datadome.co]. Do you want to 
take a stab?

> Hide key containing "secret" in web interface
> -
>
> Key: FLINK-8793
> URL: https://issues.apache.org/jira/browse/FLINK-8793
> Project: Flink
>  Issue Type: Bug
>  Components: REST
>Affects Versions: 1.4.1
>Reporter: Etienne CARRIERE
>Priority: Critical
> Fix For: 1.5.0
>
>
> Currently, we going in /jobmanager/config on the web interface, the value of 
> the key containing "password" are replaced by "" 
> When using s3 for checkpoint/savepoint configuration on an infrastructure 
> which is not on AWS (where IAM is not possible), the s3.secret-key is 
> revealed from the interface. 
> I propose the same behaviour as key with "password" for key with "secret" 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (FLINK-7219) Current allocate strategy cann‘t achieve the optimal effect with input's location

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reopened FLINK-7219:
-

reopen to fix release notes.

Please don't write comments like this in the release notes. The release notes 
should contain notes for users that want to find out what new things they need 
to consider in a release.

> Current allocate strategy cann‘t achieve the optimal effect with input's 
> location
> -
>
> Key: FLINK-7219
> URL: https://issues.apache.org/jira/browse/FLINK-7219
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.3.1
>Reporter: Sihua Zhou
>Priority: Major
>
> This is second subtask of issue 
> [FLINK-7153|https://issues.apache.org/jira/browse/FLINK-7153?filter=-2].
> Current allocate strategy can't allocate the slot optimize.  Here is the test 
> case:
> {code}
> JobVertex v1 = new JobVertex("v1", jid1);
> JobVertex v2 = new JobVertex("v2", jid2);
> SlotSharingGroup group = new SlotSharingGroup();
> v1.setSlotSharingGroup(group);
> v2.setSlotSharingGroup(group);
> v1.setParallelism(2);
> v2.setParallelism(4);
> v1.setInvokableClass(BatchTask.class);
> v2.setInvokableClass(BatchTask.class);
> v2.connectNewDataSetAsInput(v1, DistributionPattern.POINTWISE, 
> ResultPartitionType.PIPELINED_BOUNDED);
> {code}
> Currently, after allocate for v1,v2, we got a local partition and three 
> remote partition. But actually, it should be 2 local partition and 2 remote 
> partition. 
> The causes of the above problems is becuase that the current allocate 
> strategy is allocate the resource for execution one by one(if the execution 
> can allocate from SlotGroup than get it, Otherwise ask for a new one for it). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-7219) Current allocate strategy cann‘t achieve the optimal effect with input's location

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated FLINK-7219:

Release Note:   (was: fixed by till in 1.4)

> Current allocate strategy cann‘t achieve the optimal effect with input's 
> location
> -
>
> Key: FLINK-7219
> URL: https://issues.apache.org/jira/browse/FLINK-7219
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.3.1
>Reporter: Sihua Zhou
>Priority: Major
>
> This is second subtask of issue 
> [FLINK-7153|https://issues.apache.org/jira/browse/FLINK-7153?filter=-2].
> Current allocate strategy can't allocate the slot optimize.  Here is the test 
> case:
> {code}
> JobVertex v1 = new JobVertex("v1", jid1);
> JobVertex v2 = new JobVertex("v2", jid2);
> SlotSharingGroup group = new SlotSharingGroup();
> v1.setSlotSharingGroup(group);
> v2.setSlotSharingGroup(group);
> v1.setParallelism(2);
> v2.setParallelism(4);
> v1.setInvokableClass(BatchTask.class);
> v2.setInvokableClass(BatchTask.class);
> v2.connectNewDataSetAsInput(v1, DistributionPattern.POINTWISE, 
> ResultPartitionType.PIPELINED_BOUNDED);
> {code}
> Currently, after allocate for v1,v2, we got a local partition and three 
> remote partition. But actually, it should be 2 local partition and 2 remote 
> partition. 
> The causes of the above problems is becuase that the current allocate 
> strategy is allocate the resource for execution one by one(if the execution 
> can allocate from SlotGroup than get it, Otherwise ask for a new one for it). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)