[jira] [Updated] (FLINK-9199) Malfunctioning URL target in some messageheaders

[Rong Rong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rong Rong updated FLINK-9199:
-
Summary: Malfunctioning URL target in some messageheaders  (was: 
SubtaskExecutionAttemptAccumulatorsHeaders & 
SubtaskExecutionAttemptDetailsHeaders has malfunctioning URL)

> Malfunctioning URL target in some messageheaders
> 
>
> Key: FLINK-9199
> URL: https://issues.apache.org/jira/browse/FLINK-9199
> Project: Flink
>  Issue Type: Bug
>  Components: REST
>Reporter: Rong Rong
>Assignee: Rong Rong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-9199) SubtaskExecutionAttemptAccumulatorsHeaders & SubtaskExecutionAttemptDetailsHeaders has malfunctioning URL

[Rong Rong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rong Rong reassigned FLINK-9199:


Assignee: Rong Rong

> SubtaskExecutionAttemptAccumulatorsHeaders & 
> SubtaskExecutionAttemptDetailsHeaders has malfunctioning URL
> -
>
> Key: FLINK-9199
> URL: https://issues.apache.org/jira/browse/FLINK-9199
> Project: Flink
>  Issue Type: Bug
>  Components: REST
>Reporter: Rong Rong
>Assignee: Rong Rong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[Gary Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441927#comment-16441927
 ] 

Gary Yao commented on FLINK-9194:
-

[~yuqi] I configured {{jobmanager.archive.fs.dir}}, and started the History 
Server as described here: 
https://ci.apache.org/projects/flink/flink-docs-master/monitoring/historyserver.html#overview
It cannot work because the code is not there.

> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9199) SubtaskExecutionAttemptAccumulatorsHeaders & SubtaskExecutionAttemptDetailsHeaders has malfunctioning URL

[Rong Rong (JIRA)]

Rong Rong created FLINK-9199:


 Summary: SubtaskExecutionAttemptAccumulatorsHeaders & 
SubtaskExecutionAttemptDetailsHeaders has malfunctioning URL
 Key: FLINK-9199
 URL: https://issues.apache.org/jira/browse/FLINK-9199
 Project: Flink
  Issue Type: Bug
  Components: REST
Reporter: Rong Rong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[yuqi (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441846#comment-16441846
 ] 

yuqi commented on FLINK-9194:
-

[~gjy] Could not reproduce this problem with the environment you provide, more 
detail about it?

> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-9040) JobVertex#setMaxParallelism does not validate argument

[Sihua Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sihua Zhou reassigned FLINK-9040:
-

Assignee: (was: Sihua Zhou)

> JobVertex#setMaxParallelism does not validate argument
> --
>
> Key: FLINK-9040
> URL: https://issues.apache.org/jira/browse/FLINK-9040
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Priority: Minor
>
> {code}
> /**
> * Sets the maximum parallelism for the task.
> *
> * @param maxParallelism The maximum parallelism to be set. must be between 1 
> and Short.MAX_VALUE.
> */
> public void setMaxParallelism(int maxParallelism) {
>   this.maxParallelism = maxParallelism;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9040) JobVertex#setMaxParallelism does not validate argument

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441799#comment-16441799
 ] 

ASF GitHub Bot commented on FLINK-9040:
---

Github user sihuazhou commented on the issue:

https://github.com/apache/flink/pull/5825
  
closing this cause I think this maybe a no-issue.


> JobVertex#setMaxParallelism does not validate argument
> --
>
> Key: FLINK-9040
> URL: https://issues.apache.org/jira/browse/FLINK-9040
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Sihua Zhou
>Priority: Minor
>
> {code}
> /**
> * Sets the maximum parallelism for the task.
> *
> * @param maxParallelism The maximum parallelism to be set. must be between 1 
> and Short.MAX_VALUE.
> */
> public void setMaxParallelism(int maxParallelism) {
>   this.maxParallelism = maxParallelism;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5825: [FLINK-9040][local runtime] check maxParallelism in JobVe...

[sihuazhou]

Github user sihuazhou commented on the issue:

https://github.com/apache/flink/pull/5825
  
closing this cause I think this maybe a no-issue.


---

[GitHub] flink pull request #5825: [FLINK-9040][local runtime] check maxParallelism i...

[sihuazhou]

Github user sihuazhou closed the pull request at:

https://github.com/apache/flink/pull/5825


---

[jira] [Commented] (FLINK-9040) JobVertex#setMaxParallelism does not validate argument

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441800#comment-16441800
 ] 

ASF GitHub Bot commented on FLINK-9040:
---

Github user sihuazhou closed the pull request at:

https://github.com/apache/flink/pull/5825


> JobVertex#setMaxParallelism does not validate argument
> --
>
> Key: FLINK-9040
> URL: https://issues.apache.org/jira/browse/FLINK-9040
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Sihua Zhou
>Priority: Minor
>
> {code}
> /**
> * Sets the maximum parallelism for the task.
> *
> * @param maxParallelism The maximum parallelism to be set. must be between 1 
> and Short.MAX_VALUE.
> */
> public void setMaxParallelism(int maxParallelism) {
>   this.maxParallelism = maxParallelism;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (FLINK-9193) Deprecate non-well-defined output methods on DataStream

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441743#comment-16441743
 ] 

mingleizhang edited comment on FLINK-9193 at 4/18/18 1:41 AM:
--

I agree with [~twalthr] at this point. It would be better adding more comments 
on those method like those method can not support consistency guarantees 
instead of deprecating them. Like in data analysis for data cleaning, we do not 
need strong consistency guarantees and many data are drop and duplicate.


was (Author: mingleizhang):
I agree with [~twalthr] at this point. It would be better adding more comments 
on those method like those method can not support consistency guarantees 
instead of deprecating them. Like in data analysis for data cleaning, we do 
need strong consistency guarantees and many data are drop and duplicate.

> Deprecate non-well-defined output methods on DataStream
> ---
>
> Key: FLINK-9193
> URL: https://issues.apache.org/jira/browse/FLINK-9193
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 1.5.0
>
>
> Some output methods on {{DataStream}} that write text to files are not safe 
> to use in a streaming program as they have no consistency guarantees. They 
> are:
>  - {{writeAsText()}}
>  - {{writeAsCsv()}}
>  - {{writeToSocket()}}
>  - {{writeUsingOutputFormat()}}
> Along with those we should also deprecate the {{SinkFunctions}} that they use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (FLINK-9193) Deprecate non-well-defined output methods on DataStream

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441743#comment-16441743
 ] 

mingleizhang edited comment on FLINK-9193 at 4/18/18 1:40 AM:
--

I agree with [~twalthr] at this point. It would be better adding more comments 
on those method like those method can not support consistency guarantees 
instead of deprecating them. Like in data analysis for data cleaning, we do 
need strong consistency guarantees and many data are drop and duplicate.


was (Author: mingleizhang):
I agree with [~twalthr] at this point. It would be better adding more comments 
on those method like those method can not support consistency guarantees 
instead of deprecating them. Like in data analysis for data cleaning, we do 
need strong consistency guarantees.

> Deprecate non-well-defined output methods on DataStream
> ---
>
> Key: FLINK-9193
> URL: https://issues.apache.org/jira/browse/FLINK-9193
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 1.5.0
>
>
> Some output methods on {{DataStream}} that write text to files are not safe 
> to use in a streaming program as they have no consistency guarantees. They 
> are:
>  - {{writeAsText()}}
>  - {{writeAsCsv()}}
>  - {{writeToSocket()}}
>  - {{writeUsingOutputFormat()}}
> Along with those we should also deprecate the {{SinkFunctions}} that they use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9193) Deprecate non-well-defined output methods on DataStream

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441743#comment-16441743
 ] 

mingleizhang commented on FLINK-9193:
-

I agree with [~twalthr] at this point. It would be better adding more comments 
on those method like those method can not support consistency guarantees 
instead of deprecating them. Like in data analysis for data cleaning, we do 
need strong consistency guarantees.

> Deprecate non-well-defined output methods on DataStream
> ---
>
> Key: FLINK-9193
> URL: https://issues.apache.org/jira/browse/FLINK-9193
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 1.5.0
>
>
> Some output methods on {{DataStream}} that write text to files are not safe 
> to use in a streaming program as they have no consistency guarantees. They 
> are:
>  - {{writeAsText()}}
>  - {{writeAsCsv()}}
>  - {{writeToSocket()}}
>  - {{writeUsingOutputFormat()}}
> Along with those we should also deprecate the {{SinkFunctions}} that they use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9187) add prometheus pushgateway reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441740#comment-16441740
 ] 

ASF GitHub Bot commented on FLINK-9187:
---

Github user lamber-ken commented on the issue:

https://github.com/apache/flink/pull/5857
  
cc @tillrohrmann : If you have time, please review this PR, thanks.


> add prometheus pushgateway reporter
> ---
>
> Key: FLINK-9187
> URL: https://issues.apache.org/jira/browse/FLINK-9187
> Project: Flink
>  Issue Type: New Feature
>  Components: Metrics
>Affects Versions: 1.4.2
>Reporter: lamber-ken
>Priority: Minor
>  Labels: features
> Fix For: 1.5.0
>
>
> make flink system can send metrics to prometheus via pushgateway.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5857: [FLINK-9187][METRICS] add prometheus pushgateway reporter

[lamber-ken]

Github user lamber-ken commented on the issue:

https://github.com/apache/flink/pull/5857
  
cc @tillrohrmann : If you have time, please review this PR, thanks.


---

[GitHub] flink pull request #5864: [FLINK-8661] Replace Collections.EMPTY_MAP with Co...

[zhangminglei]

GitHub user zhangminglei opened a pull request:

https://github.com/apache/flink/pull/5864

[FLINK-8661] Replace Collections.EMPTY_MAP with Collections.emptyMap()


## What is the purpose of the change

The use of Collections.EMPTY_SET and Collections.EMPTY_MAP often causes 
unchecked assignment. It should be replaced with Collections.emptySet() and 
Collections.emptyMap() .



## Brief change log
Change Collections.EMPTY_MAP and Collections.EMPTY_LIST to 
Collections.emptyList() & Collections.emptyMap().

## Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhangminglei/flink flink-8661

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5864.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5864


commit f021149aa73576a9a9fd05ba00fcf7e5d1901318
Author: zhangminglei 
Date:   2018-04-18T01:24:57Z

[FLINK-8661] Replace Collections.EMPTY_MAP with Collections.emptyMap()




---

[jira] [Commented] (FLINK-8661) Replace Collections.EMPTY_MAP with Collections.emptyMap()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441736#comment-16441736
 ] 

ASF GitHub Bot commented on FLINK-8661:
---

GitHub user zhangminglei opened a pull request:

https://github.com/apache/flink/pull/5864

[FLINK-8661] Replace Collections.EMPTY_MAP with Collections.emptyMap()


## What is the purpose of the change

The use of Collections.EMPTY_SET and Collections.EMPTY_MAP often causes 
unchecked assignment. It should be replaced with Collections.emptySet() and 
Collections.emptyMap() .



## Brief change log
Change Collections.EMPTY_MAP and Collections.EMPTY_LIST to 
Collections.emptyList() & Collections.emptyMap().

## Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhangminglei/flink flink-8661

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5864.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5864


commit f021149aa73576a9a9fd05ba00fcf7e5d1901318
Author: zhangminglei 
Date:   2018-04-18T01:24:57Z

[FLINK-8661] Replace Collections.EMPTY_MAP with Collections.emptyMap()




> Replace Collections.EMPTY_MAP with Collections.emptyMap()
> -
>
> Key: FLINK-8661
> URL: https://issues.apache.org/jira/browse/FLINK-8661
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> The use of Collections.EMPTY_SET and Collections.EMPTY_MAP often causes 
> unchecked assignment. It should be replaced with Collections.emptySet() and 
> Collections.emptyMap() .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8661) Replace Collections.EMPTY_MAP with Collections.emptyMap()

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441734#comment-16441734
 ] 

mingleizhang commented on FLINK-8661:
-

Sorry for later since my friend wants to get this ticket. But I think he is too 
busy to do it. So, I will give a PR soon.

> Replace Collections.EMPTY_MAP with Collections.emptyMap()
> -
>
> Key: FLINK-8661
> URL: https://issues.apache.org/jira/browse/FLINK-8661
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> The use of Collections.EMPTY_SET and Collections.EMPTY_MAP often causes 
> unchecked assignment. It should be replaced with Collections.emptySet() and 
> Collections.emptyMap() .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-9048) LocalFlinkMiniClusterITCase#testLocalFlinkMiniClusterWithMultipleTaskManagers sometimes fails

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-9048:
--
Description: 
As of commit e0bc37bef69f5376d03214578e9b95816add661b, I got the following :

{code}
testLocalFlinkMiniClusterWithMultipleTaskManagers(org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase)
  Time elapsed: 41.681 sec  <<< FAILURE!
java.lang.AssertionError: Thread 
Thread[ForkJoinPool.commonPool-worker-25,5,main] was started by the mini 
cluster, but not shut down
  at org.junit.Assert.fail(Assert.java:88)
  at 
org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase.testLocalFlinkMiniClusterWithMultipleTaskManagers(LocalFlinkMiniClusterITCase.java:174)
{code}

  was:
As of commit e0bc37bef69f5376d03214578e9b95816add661b, I got the following :
{code}
testLocalFlinkMiniClusterWithMultipleTaskManagers(org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase)
  Time elapsed: 41.681 sec  <<< FAILURE!
java.lang.AssertionError: Thread 
Thread[ForkJoinPool.commonPool-worker-25,5,main] was started by the mini 
cluster, but not shut down
  at org.junit.Assert.fail(Assert.java:88)
  at 
org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase.testLocalFlinkMiniClusterWithMultipleTaskManagers(LocalFlinkMiniClusterITCase.java:174)
{code}


> LocalFlinkMiniClusterITCase#testLocalFlinkMiniClusterWithMultipleTaskManagers 
> sometimes fails
> -
>
> Key: FLINK-9048
> URL: https://issues.apache.org/jira/browse/FLINK-9048
> Project: Flink
>  Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
>
> As of commit e0bc37bef69f5376d03214578e9b95816add661b, I got the following :
> {code}
> testLocalFlinkMiniClusterWithMultipleTaskManagers(org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase)
>   Time elapsed: 41.681 sec  <<< FAILURE!
> java.lang.AssertionError: Thread 
> Thread[ForkJoinPool.commonPool-worker-25,5,main] was started by the mini 
> cluster, but not shut down
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase.testLocalFlinkMiniClusterWithMultipleTaskManagers(LocalFlinkMiniClusterITCase.java:174)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8335) Upgrade hbase connector dependency to 1.4.3

[mingleizhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mingleizhang updated FLINK-8335:

Fix Version/s: 1.6.0

> Upgrade hbase connector dependency to 1.4.3
> ---
>
> Key: FLINK-8335
> URL: https://issues.apache.org/jira/browse/FLINK-8335
> Project: Flink
>  Issue Type: Improvement
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
> Fix For: 1.6.0
>
>
> hbase 1.4.3 has been released.
> 1.4.0 shows speed improvement over previous 1.x releases.
> http://search-hadoop.com/m/HBase/YGbbBxedD1Mnm8t?subj=Re+VOTE+The+second+HBase+1+4+0+release+candidate+RC1+is+available
> This issue is to upgrade the dependency to 1.4.3



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8335) Upgrade hbase connector dependency to 1.4.3

[mingleizhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mingleizhang updated FLINK-8335:

Description: 
hbase 1.4.3 has been released.

1.4.0 shows speed improvement over previous 1.x releases.

http://search-hadoop.com/m/HBase/YGbbBxedD1Mnm8t?subj=Re+VOTE+The+second+HBase+1+4+0+release+candidate+RC1+is+available

This issue is to upgrade the dependency to 1.4.3

  was:
hbase 1.4.1 has been released.

1.4.0 shows speed improvement over previous 1.x releases.

http://search-hadoop.com/m/HBase/YGbbBxedD1Mnm8t?subj=Re+VOTE+The+second+HBase+1+4+0+release+candidate+RC1+is+available

This issue is to upgrade the dependency to 1.4.1


> Upgrade hbase connector dependency to 1.4.3
> ---
>
> Key: FLINK-8335
> URL: https://issues.apache.org/jira/browse/FLINK-8335
> Project: Flink
>  Issue Type: Improvement
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> hbase 1.4.3 has been released.
> 1.4.0 shows speed improvement over previous 1.x releases.
> http://search-hadoop.com/m/HBase/YGbbBxedD1Mnm8t?subj=Re+VOTE+The+second+HBase+1+4+0+release+candidate+RC1+is+available
> This issue is to upgrade the dependency to 1.4.3



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8335) Upgrade hbase connector dependency to 1.4.3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441727#comment-16441727
 ] 

ASF GitHub Bot commented on FLINK-8335:
---

Github user zhangminglei commented on the issue:

https://github.com/apache/flink/pull/5488
  
Okay  


> Upgrade hbase connector dependency to 1.4.3
> ---
>
> Key: FLINK-8335
> URL: https://issues.apache.org/jira/browse/FLINK-8335
> Project: Flink
>  Issue Type: Improvement
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> hbase 1.4.1 has been released.
> 1.4.0 shows speed improvement over previous 1.x releases.
> http://search-hadoop.com/m/HBase/YGbbBxedD1Mnm8t?subj=Re+VOTE+The+second+HBase+1+4+0+release+candidate+RC1+is+available
> This issue is to upgrade the dependency to 1.4.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5488: [FLINK-8335] [hbase] Upgrade hbase connector dependency t...

[zhangminglei]

Github user zhangminglei commented on the issue:

https://github.com/apache/flink/pull/5488
  
Okay 👍 


---

[jira] [Commented] (FLINK-9138) Enhance BucketingSink to also flush data by time interval

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441716#comment-16441716
 ] 

ASF GitHub Bot commented on FLINK-9138:
---

Github user glaksh100 commented on the issue:

https://github.com/apache/flink/pull/5860
  
@aljoscha @fhueske Can you please take a look? 


> Enhance BucketingSink to also flush data by time interval
> -
>
> Key: FLINK-9138
> URL: https://issues.apache.org/jira/browse/FLINK-9138
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Affects Versions: 1.4.2
>Reporter: Narayanan Arunachalam
>Priority: Major
>
> BucketingSink now supports flushing data to the file system by size limit and 
> by period of inactivity. It will be useful to also flush data by a specified 
> time period. This way, the data will be written out when write throughput is 
> low but there is no significant time period gaps between the writes. This 
> reduces ETA for the data in the file system and should help move the 
> checkpoints faster as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5860: [FLINK-9138][filesystem-connectors] Implement time based ...

[glaksh100]

Github user glaksh100 commented on the issue:

https://github.com/apache/flink/pull/5860
  
@aljoscha @fhueske Can you please take a look? 


---

[jira] [Comment Edited] (FLINK-6105) Properly handle InterruptedException in HadoopInputFormatBase

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307281#comment-16307281
 ] 

Ted Yu edited comment on FLINK-6105 at 4/17/18 10:53 PM:
-

In 
flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/RollingSink.java
 :

{code}
  try {
Thread.sleep(500);
  } catch (InterruptedException e1) {
// ignore it
  }
{code}

Interrupt status should be restored, or throw InterruptedIOException .


was (Author: yuzhih...@gmail.com):
In 
flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/RollingSink.java
 :
{code}
  try {
Thread.sleep(500);
  } catch (InterruptedException e1) {
// ignore it
  }
{code}

Interrupt status should be restored, or throw InterruptedIOException .

> Properly handle InterruptedException in HadoopInputFormatBase
> -
>
> Key: FLINK-6105
> URL: https://issues.apache.org/jira/browse/FLINK-6105
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Major
>
> When catching InterruptedException, we should throw InterruptedIOException 
> instead of IOException.
> The following example is from HadoopInputFormatBase :
> {code}
> try {
>   splits = this.mapreduceInputFormat.getSplits(jobContext);
> } catch (InterruptedException e) {
>   throw new IOException("Could not get Splits.", e);
> }
> {code}
> There may be other places where IOE is thrown.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7525) Add config option to disable Cancel functionality on UI

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441630#comment-16441630
 ] 

Ted Yu commented on FLINK-7525:
---

Hopefully FLIP-6 would be released soon.

> Add config option to disable Cancel functionality on UI
> ---
>
> Key: FLINK-7525
> URL: https://issues.apache.org/jira/browse/FLINK-7525
> Project: Flink
>  Issue Type: Improvement
>  Components: Web Client, Webfrontend
>Reporter: Ted Yu
>Priority: Major
>
> In this email thread 
> http://search-hadoop.com/m/Flink/VkLeQlf0QOnc7YA?subj=Security+Control+of+running+Flink+Jobs+on+Flink+UI
>  , Raja was asking for a way to control how users cancel Job(s).
> Robert proposed adding a config option which disables the Cancel 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (FLINK-7795) Utilize error-prone to discover common coding mistakes

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345955#comment-16345955
 ] 

Ted Yu edited comment on FLINK-7795 at 4/17/18 10:45 PM:
-

error-prone has JDK 8 dependency .


was (Author: yuzhih...@gmail.com):
error-prone has JDK 8 dependency.

> Utilize error-prone to discover common coding mistakes
> --
>
> Key: FLINK-7795
> URL: https://issues.apache.org/jira/browse/FLINK-7795
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Reporter: Ted Yu
>Priority: Major
>
> http://errorprone.info/ is a tool which detects common coding mistakes.
> We should incorporate into Flink build process.
> Here are the dependencies:
> {code}
> 
>   com.google.errorprone
>   error_prone_annotation
>   ${error-prone.version}
>   provided
> 
> 
>   
>   com.google.auto.service
>   auto-service
>   1.0-rc3
>   true
> 
> 
>   com.google.errorprone
>   error_prone_check_api
>   ${error-prone.version}
>   provided
>   
> 
>   com.google.code.findbugs
>   jsr305
> 
>   
> 
> 
>   com.google.errorprone
>   javac
>   9-dev-r4023-3
>   provided
> 
>   
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8661) Replace Collections.EMPTY_MAP with Collections.emptyMap()

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441586#comment-16441586
 ] 

Chesnay Schepler commented on FLINK-8661:
-

I don't see a commit for it, so I'd keep it open.

> Replace Collections.EMPTY_MAP with Collections.emptyMap()
> -
>
> Key: FLINK-8661
> URL: https://issues.apache.org/jira/browse/FLINK-8661
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> The use of Collections.EMPTY_SET and Collections.EMPTY_MAP often causes 
> unchecked assignment. It should be replaced with Collections.emptySet() and 
> Collections.emptyMap() .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7897) Consider using nio.Files for file deletion in TransientBlobCleanupTask

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441585#comment-16441585
 ] 

Ted Yu commented on FLINK-7897:
---

lgtm

> Consider using nio.Files for file deletion in TransientBlobCleanupTask
> --
>
> Key: FLINK-7897
> URL: https://issues.apache.org/jira/browse/FLINK-7897
> Project: Flink
>  Issue Type: Improvement
>  Components: Local Runtime
>Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Minor
>
> nio.Files#delete() provides better clue as to why the deletion may fail:
> https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#delete(java.nio.file.Path)
> Depending on the potential exception (FileNotFound), the call to 
> localFile.exists() may be skipped.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (FLINK-7917) The return of taskInformationOrBlobKey should be placed inside synchronized in ExecutionJobVertex

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432712#comment-16432712
 ] 

Ted Yu edited comment on FLINK-7917 at 4/17/18 10:15 PM:
-

+1


was (Author: yuzhih...@gmail.com):
lgtm

> The return of taskInformationOrBlobKey should be placed inside synchronized 
> in ExecutionJobVertex
> -
>
> Key: FLINK-7917
> URL: https://issues.apache.org/jira/browse/FLINK-7917
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Minor
>
> Currently in ExecutionJobVertex#getTaskInformationOrBlobKey:
> {code}
> }
> return taskInformationOrBlobKey;
> {code}
> The return should be placed inside synchronized block.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8661) Replace Collections.EMPTY_MAP with Collections.emptyMap()

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441581#comment-16441581
 ] 

Ted Yu commented on FLINK-8661:
---

Should this issue be resolved ?

> Replace Collections.EMPTY_MAP with Collections.emptyMap()
> -
>
> Key: FLINK-8661
> URL: https://issues.apache.org/jira/browse/FLINK-8661
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> The use of Collections.EMPTY_SET and Collections.EMPTY_MAP often causes 
> unchecked assignment. It should be replaced with Collections.emptySet() and 
> Collections.emptyMap() .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8985) End-to-end test: CLI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441481#comment-16441481
 ] 

ASF GitHub Bot commented on FLINK-8985:
---

GitHub user walterddr opened a pull request:

https://github.com/apache/flink/pull/5863

[FLINK-8985][e2etest] initial support for End-to-end CLI test, excluding 
YARN test

## What is the purpose of the change

Adding end to end test for CLI APIs.

## Brief change log

Added test_cli_api.sh script to test combinations of CLI commands listed in 
the doc section of Flink. Including:
  - Start up command sets (run)
  - Operational command sets (list/info/cancel)
  - Savepoint command sets (savepoint)

## Verifying this change

This is a test

## Does this pull request potentially affect one of the following parts:

No

## Documentation

No

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/walterddr/flink FLINK-8985

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5863.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5863


commit 5f36ee5d4dcbb60a29a413fd29cfaaa69f7e8a47
Author: Rong Rong 
Date:   2018-04-17T20:45:23Z

initial commit to support CLI test, excluding YARN test




> End-to-end test: CLI
> 
>
> Key: FLINK-8985
> URL: https://issues.apache.org/jira/browse/FLINK-8985
> Project: Flink
>  Issue Type: Sub-task
>  Components: Client, Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Rong Rong
>Priority: Major
>
> We should an end-to-end test which verifies that all client commands are 
> working correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5863: [FLINK-8985][e2etest] initial support for End-to-e...

[walterddr]

GitHub user walterddr opened a pull request:

https://github.com/apache/flink/pull/5863

[FLINK-8985][e2etest] initial support for End-to-end CLI test, excluding 
YARN test

## What is the purpose of the change

Adding end to end test for CLI APIs.

## Brief change log

Added test_cli_api.sh script to test combinations of CLI commands listed in 
the doc section of Flink. Including:
  - Start up command sets (run)
  - Operational command sets (list/info/cancel)
  - Savepoint command sets (savepoint)

## Verifying this change

This is a test

## Does this pull request potentially affect one of the following parts:

No

## Documentation

No

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/walterddr/flink FLINK-8985

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5863.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5863


commit 5f36ee5d4dcbb60a29a413fd29cfaaa69f7e8a47
Author: Rong Rong 
Date:   2018-04-17T20:45:23Z

initial commit to support CLI test, excluding YARN test




---

[GitHub] flink issue #5488: [FLINK-8335] [hbase] Upgrade hbase connector dependency t...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/5488
  
All right, trying to merge this for 1.6


---

[jira] [Commented] (FLINK-8335) Upgrade hbase connector dependency to 1.4.3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441438#comment-16441438
 ] 

ASF GitHub Bot commented on FLINK-8335:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/5488
  
All right, trying to merge this for 1.6


> Upgrade hbase connector dependency to 1.4.3
> ---
>
> Key: FLINK-8335
> URL: https://issues.apache.org/jira/browse/FLINK-8335
> Project: Flink
>  Issue Type: Improvement
>  Components: Batch Connectors and Input/Output Formats
>Reporter: Ted Yu
>Assignee: mingleizhang
>Priority: Minor
>
> hbase 1.4.1 has been released.
> 1.4.0 shows speed improvement over previous 1.x releases.
> http://search-hadoop.com/m/HBase/YGbbBxedD1Mnm8t?subj=Re+VOTE+The+second+HBase+1+4+0+release+candidate+RC1+is+available
> This issue is to upgrade the dependency to 1.4.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-6924) ADD LOG(X) supported in TableAPI

[Fabian Hueske (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fabian Hueske closed FLINK-6924.

   Resolution: Fixed
Fix Version/s: 1.6.0

Implemented for 1.6.0 with d38695b8e99d62777b2bca964a5c487a67e42331

> ADD LOG(X) supported in TableAPI
> 
>
> Key: FLINK-6924
> URL: https://issues.apache.org/jira/browse/FLINK-6924
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: buptljy
>Priority: Major
>  Labels: starter
> Fix For: 1.6.0
>
>
> See FLINK-6891 for detail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8366) Use Row instead of String as key when process upsert results

[Fabian Hueske (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fabian Hueske closed FLINK-8366.

   Resolution: Fixed
Fix Version/s: 1.5.0

Fixed on master with 3adc21d489d78cd34748f2132e4e7659f65a33e4
Fixed for 1.5.0 with bea431f131c52f636881e86dee2fb195ab56db9e

> Use Row instead of String as key when process upsert results
> 
>
> Key: FLINK-8366
> URL: https://issues.apache.org/jira/browse/FLINK-8366
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
> Fix For: 1.5.0
>
>
> In {{TableSinkITCase.upsertResults()}} function, we use String as key to 
> upsert results. This will make (1,11) and (11,1) have the same key (i.e., 
> 111).
> This bugfix use Row instead of String to avoid the String problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5638: [FLINK-6924][table]ADD LOG(X) supported in TableAP...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5638


---

[jira] [Commented] (FLINK-6924) ADD LOG(X) supported in TableAPI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441400#comment-16441400
 ] 

ASF GitHub Bot commented on FLINK-6924:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5638


> ADD LOG(X) supported in TableAPI
> 
>
> Key: FLINK-6924
> URL: https://issues.apache.org/jira/browse/FLINK-6924
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: buptljy
>Priority: Major
>  Labels: starter
>
> See FLINK-6891 for detail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5244: [FLINK-8366] [table] Use Row instead of String as ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5244


---

[jira] [Commented] (FLINK-8366) Use Row instead of String as key when process upsert results

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441399#comment-16441399
 ] 

ASF GitHub Bot commented on FLINK-8366:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5244


> Use Row instead of String as key when process upsert results
> 
>
> Key: FLINK-8366
> URL: https://issues.apache.org/jira/browse/FLINK-8366
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
>
> In {{TableSinkITCase.upsertResults()}} function, we use String as key to 
> upsert results. This will make (1,11) and (11,1) have the same key (i.e., 
> 111).
> This bugfix use Row instead of String to avoid the String problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9158) Set default FixedRestartDelayStrategy delay to 0s

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441397#comment-16441397
 ] 

Stephan Ewen commented on FLINK-9158:
-

I think there is no reason for a default delay. It just makes the out of the 
box experience a bit worse, because it looks like Flink needs very long to 
recover.

> Set default FixedRestartDelayStrategy delay to 0s
> -
>
> Key: FLINK-9158
> URL: https://issues.apache.org/jira/browse/FLINK-9158
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Sihua Zhou
>Priority: Blocker
> Fix For: 1.5.0
>
>
> Set default FixedRestartDelayStrategy delay to 0s.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9158) Set default FixedRestartDelayStrategy delay to 0s

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441391#comment-16441391
 ] 

ASF GitHub Bot commented on FLINK-9158:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/5839
  
Looks good, thanks!
Merging...


> Set default FixedRestartDelayStrategy delay to 0s
> -
>
> Key: FLINK-9158
> URL: https://issues.apache.org/jira/browse/FLINK-9158
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Sihua Zhou
>Priority: Blocker
> Fix For: 1.5.0
>
>
> Set default FixedRestartDelayStrategy delay to 0s.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5839: [FLINK-9158][Distributed Coordination] Set default FixedR...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/5839
  
Looks good, thanks!
Merging...


---

[jira] [Created] (FLINK-9198) Improve error messages in AbstractDeserializationSchema for type extraction

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-9198:
---

 Summary: Improve error messages in AbstractDeserializationSchema 
for type extraction
 Key: FLINK-9198
 URL: https://issues.apache.org/jira/browse/FLINK-9198
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.4.2
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.5.0


User feedback: When type extraction fails in the 
{{AbstractDeserializationSchema}}, the error message does not explain fully how 
to fix this.

I suggest to improve the error message and add some convenience constructors to 
directly pass TypeInformation when needed.

We can also simplify the class a bit, because TypeInformation needs no longer 
be dropped during serialization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9197) Improve error message for TypyInformation and TypeHint with generics

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-9197:
---

 Summary: Improve error message for TypyInformation and TypeHint 
with generics
 Key: FLINK-9197
 URL: https://issues.apache.org/jira/browse/FLINK-9197
 Project: Flink
  Issue Type: Improvement
  Components: Core
Affects Versions: 1.4.2
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.5.0


User feedback: When using a {{TypeHint}} with a generic type variable, the 
error message could be better. Similarly, when using 
{{TypeInformation.of(Tuple2.class)}}, the error message should refer the user 
to the TypeHint method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9196) YARN: Flink binaries are not deleted from HDFS after cluster shutdown

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441248#comment-16441248
 ] 

Stephan Ewen commented on FLINK-9196:
-

I think this may explain parts of the error described here: 
https://lists.apache.org/thread.html/73b652d2f4ba1167ff8166dc8260cf3458df562c630f8f891eb3@%3Cuser.flink.apache.org%3E

> YARN: Flink binaries are not deleted from HDFS after cluster shutdown
> -
>
> Key: FLINK-9196
> URL: https://issues.apache.org/jira/browse/FLINK-9196
> Project: Flink
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.5.0
>Reporter: Gary Yao
>Priority: Blocker
>  Labels: flip-6
>
> When deploying on YARN in flip6 mode, the Flink binaries are not deleted from 
> HDFS after the cluster shuts down.
> *Steps to reproduce*
> # Submit job in YARN job mode, non-detached:
> {noformat} HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster 
> -yjm 2048 -ytm 2048 -yn 2  ./examples/streaming/WordCount.jar {noformat}
> # Check contents of {{/user/hadoop/.flink/}} on HDFS after 
> job is finished:
> {noformat}
> [hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls 
> /user/hadoop/.flink/application_1523966184826_0016
> Found 6 items
> -rw-r--r--   1 hadoop hadoop583 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml
> -rw-r--r--   1 hadoop hadoop332 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp
> -rw-r--r--   1 hadoop hadoop   89779342 2018-04-02 17:08 
> /user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar
> drwxrwxrwx   - hadoop hadoop  0 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/lib
> -rw-r--r--   1 hadoop hadoop   1939 2018-04-02 15:37 
> /user/hadoop/.flink/application_1523966184826_0016/log4j.properties
> -rw-r--r--   1 hadoop hadoop   2331 2018-04-02 15:37 
> /user/hadoop/.flink/application_1523966184826_0016/logback.xml
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-9194) Finished jobs are not archived to HistoryServer

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler reassigned FLINK-9194:
---

Assignee: Chesnay Schepler  (was: yuqi)

> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-2435) Add support for custom CSV field parsers

[Dmitrii Kober (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441211#comment-16441211
 ] 

Dmitrii Kober commented on FLINK-2435:
--

Hello [~fhueske]. Thank you for the review! I have updated the code and raised 
a new pull-request for the first couple of comments:
 - reverting all unnecessary changes such as changing import order or white 
space changes
 - merging all commits into a single commit

Could you please help with clarifying the last two? 
 - check if the changes to `TupleTypeInfo` are required. _This change is done 
to let a user-defined class be a part of a Tuple instance (as far as a tuple is 
a fixed-length 'value container'), similar to an 'unbounded' Row). Otherwise, 
only Basic and BasicValue types could be used._ 
 - check if we can add this feature without adding a dependency to 
`flink-java`. _Currently, CsvReader class resides in 'flink-java' package. Do 
you mean that this class should be migrated to 'flink-core'_ ?

> Add support for custom CSV field parsers
> 
>
> Key: FLINK-2435
> URL: https://issues.apache.org/jira/browse/FLINK-2435
> Project: Flink
>  Issue Type: New Feature
>  Components: DataSet API
>Affects Versions: 0.10.0
>Reporter: Fabian Hueske
>Assignee: Dmitrii Kober
>Priority: Minor
> Fix For: 1.0.0
>
>
> The {{CSVInputFormats}} have only {{FieldParsers}} for Java's primitive types 
> (byte, short, int, long, float, double, boolean, String).
> It would be good to add support for CSV field parsers for custom data types 
> which can be registered in a {{CSVReader}}. 
> We could offer two interfaces for field parsers.
> 1. The regular low-level {{FieldParser}} which operates on a byte array and 
> offsets.
> 2. A {{StringFieldParser}} which operates on a String that has been extracted 
> by a {{StringParser}} before. This interface will be easier to implement but 
> less efficient.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8955) Port ClassLoaderITCase to flip6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441190#comment-16441190
 ] 

ASF GitHub Bot commented on FLINK-8955:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5780
  
checkstyle  


> Port ClassLoaderITCase to flip6
> ---
>
> Key: FLINK-8955
> URL: https://issues.apache.org/jira/browse/FLINK-8955
> Project: Flink
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5780: [FLINK-8955][tests] Port ClassLoaderITCase to flip6

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5780
  
checkstyle 😡 


---

[jira] [Commented] (FLINK-9113) Data loss in BucketingSink when writing to local filesystem

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441179#comment-16441179
 ] 

ASF GitHub Bot commented on FLINK-9113:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/5861
  
It seems that for Hadoop 2.8.3 truncating is supported for the raw local 
filesystems. I will need to adapt the test for that.


> Data loss in BucketingSink when writing to local filesystem
> ---
>
> Key: FLINK-9113
> URL: https://issues.apache.org/jira/browse/FLINK-9113
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Blocker
> Fix For: 1.5.0
>
>
> For local filesystems, it is not guaranteed that the data is flushed to disk 
> during checkpointing. This leads to data loss in cases of TaskManager 
> failures when writing to a local filesystem 
> {{org.apache.hadoop.fs.LocalFileSystem}}. The {{flush()}} method returns a 
> written length but the data is not written into the file (thus the valid 
> length might be greater than the actual file size). {{hsync}} and {{hflush}} 
> have no effect either.
> It seems that this behavior won't be fixed in the near future: 
> https://issues.apache.org/jira/browse/HADOOP-7844
> One solution would be to call {{close()}} on a checkpoint for local 
> filesystems, even though this would lead to performance decrease. If we don't 
> fix this issue, we should at least add proper documentation for it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-4809) Operators should tolerate checkpoint failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441180#comment-16441180
 ] 

ASF GitHub Bot commented on FLINK-4809:
---

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/4883#discussion_r182155763
  
--- Diff: docs/dev/stream/state/checkpointing.md ---
@@ -118,6 +120,9 @@ 
env.getCheckpointConfig.setMinPauseBetweenCheckpoints(500)
 // checkpoints have to complete within one minute, or are discarded
 env.getCheckpointConfig.setCheckpointTimeout(6)
 
+// prevent the tasks from failing if an error happens in their 
checkpointing, the checkpoint will just be declined.
+env.getCheckpointConfig.setFailTasksOnCheckpointingErrors(false)
--- End diff --

This line is missing from the Java tab.


> Operators should tolerate checkpoint failures
> -
>
> Key: FLINK-4809
> URL: https://issues.apache.org/jira/browse/FLINK-4809
> Project: Flink
>  Issue Type: Sub-task
>  Components: State Backends, Checkpointing
>Reporter: Stephan Ewen
>Assignee: Stefan Richter
>Priority: Major
> Fix For: 1.5.0
>
>
> Operators should try/catch exceptions in the synchronous and asynchronous 
> part of the checkpoint and send a {{DeclineCheckpoint}} message as a result.
> The decline message should have the failure cause attached to it.
> The checkpoint barrier should be sent anyways as a first step before 
> attempting to make a state checkpoint, to make sure that downstream operators 
> do not block in alignment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5861: [FLINK-9113] [connectors] Use raw local file system for b...

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/5861
  
It seems that for Hadoop 2.8.3 truncating is supported for the raw local 
filesystems. I will need to adapt the test for that.


---

[GitHub] flink pull request #4883: [FLINK-4809] Operators should tolerate checkpoint ...

[rmetzger]

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/4883#discussion_r182155763
  
--- Diff: docs/dev/stream/state/checkpointing.md ---
@@ -118,6 +120,9 @@ 
env.getCheckpointConfig.setMinPauseBetweenCheckpoints(500)
 // checkpoints have to complete within one minute, or are discarded
 env.getCheckpointConfig.setCheckpointTimeout(6)
 
+// prevent the tasks from failing if an error happens in their 
checkpointing, the checkpoint will just be declined.
+env.getCheckpointConfig.setFailTasksOnCheckpointingErrors(false)
--- End diff --

This line is missing from the Java tab.


---

[jira] [Commented] (FLINK-2435) Add support for custom CSV field parsers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441175#comment-16441175
 ] 

ASF GitHub Bot commented on FLINK-2435:
---

GitHub user DmitryKober opened a pull request:

https://github.com/apache/flink/pull/5862

[FLINK-2435] User-defined types in CsvReader

[FLINK-2435] Provides the capability of specifying user-defined types in 
Tuple and Row 'containers' while reading from a csv file


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/DmitryKober/flink feature/flink-2435

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5862.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5862


commit 08d76d493c332f624ffae2f1794542968ce47a62
Author: Dmitrii_Kober 
Date:   2018-04-17T16:47:50Z

[FLINK-2435] Provide the capability of specifying user-defined types in 
Tuple and Row 'containers' while reading from a csv file




> Add support for custom CSV field parsers
> 
>
> Key: FLINK-2435
> URL: https://issues.apache.org/jira/browse/FLINK-2435
> Project: Flink
>  Issue Type: New Feature
>  Components: DataSet API
>Affects Versions: 0.10.0
>Reporter: Fabian Hueske
>Assignee: Dmitrii Kober
>Priority: Minor
> Fix For: 1.0.0
>
>
> The {{CSVInputFormats}} have only {{FieldParsers}} for Java's primitive types 
> (byte, short, int, long, float, double, boolean, String).
> It would be good to add support for CSV field parsers for custom data types 
> which can be registered in a {{CSVReader}}. 
> We could offer two interfaces for field parsers.
> 1. The regular low-level {{FieldParser}} which operates on a byte array and 
> offsets.
> 2. A {{StringFieldParser}} which operates on a String that has been extracted 
> by a {{StringParser}} before. This interface will be easier to implement but 
> less efficient.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5862: [FLINK-2435] User-defined types in CsvReader

[DmitryKober]

GitHub user DmitryKober opened a pull request:

https://github.com/apache/flink/pull/5862

[FLINK-2435] User-defined types in CsvReader

[FLINK-2435] Provides the capability of specifying user-defined types in 
Tuple and Row 'containers' while reading from a csv file


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/DmitryKober/flink feature/flink-2435

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5862.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5862


commit 08d76d493c332f624ffae2f1794542968ce47a62
Author: Dmitrii_Kober 
Date:   2018-04-17T16:47:50Z

[FLINK-2435] Provide the capability of specifying user-defined types in 
Tuple and Row 'containers' while reading from a csv file




---

[jira] [Commented] (FLINK-2435) Add support for custom CSV field parsers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441157#comment-16441157
 ] 

ASF GitHub Bot commented on FLINK-2435:
---

Github user DmitryKober closed the pull request at:

https://github.com/apache/flink/pull/5835


> Add support for custom CSV field parsers
> 
>
> Key: FLINK-2435
> URL: https://issues.apache.org/jira/browse/FLINK-2435
> Project: Flink
>  Issue Type: New Feature
>  Components: DataSet API
>Affects Versions: 0.10.0
>Reporter: Fabian Hueske
>Assignee: Dmitrii Kober
>Priority: Minor
> Fix For: 1.0.0
>
>
> The {{CSVInputFormats}} have only {{FieldParsers}} for Java's primitive types 
> (byte, short, int, long, float, double, boolean, String).
> It would be good to add support for CSV field parsers for custom data types 
> which can be registered in a {{CSVReader}}. 
> We could offer two interfaces for field parsers.
> 1. The regular low-level {{FieldParser}} which operates on a byte array and 
> offsets.
> 2. A {{StringFieldParser}} which operates on a String that has been extracted 
> by a {{StringParser}} before. This interface will be easier to implement but 
> less efficient.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5835: [FLINK-2435] Extending CsvReader capabilities: it ...

[DmitryKober]

Github user DmitryKober closed the pull request at:

https://github.com/apache/flink/pull/5835


---

[jira] [Assigned] (FLINK-9194) Finished jobs are not archived to HistoryServer

[yuqi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yuqi reassigned FLINK-9194:
---

Assignee: yuqi

> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: yuqi
>Priority: Blocker
>  Labels: flip-6
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8910) Introduce automated end-to-end test for local recovery (including sticky scheduling)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441060#comment-16441060
 ] 

ASF GitHub Bot commented on FLINK-8910:
---

Github user StefanRRichter commented on a diff in the pull request:

https://github.com/apache/flink/pull/5676#discussion_r182126796
  
--- Diff: 
flink-end-to-end-tests/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java
 ---
@@ -0,0 +1,451 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.tests;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.utils.ParameterTool;
+import org.apache.flink.contrib.streaming.state.RocksDBStateBackend;
+import org.apache.flink.runtime.state.CheckpointListener;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.PrintSinkFunction;
+import org.apache.flink.streaming.api.functions.source.RichSourceFunction;
+import org.apache.flink.streaming.api.operators.StreamingRuntimeContext;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Automatic end-to-end test for local recovery (including sticky 
allocation).
+ */
+public class StickyAllocationAndLocalRecoveryTestJob {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(StickyAllocationAndLocalRecoveryTestJob.class);
+
+   public static void main(String[] args) throws Exception {
+
+   final ParameterTool pt = ParameterTool.fromArgs(args);
+
+   final StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
+
+   env.setParallelism(pt.getInt("parallelism", 1));
--- End diff --

 


> Introduce automated end-to-end test for local recovery (including sticky 
> scheduling)
> 
>
> Key: FLINK-8910
> URL: https://issues.apache.org/jira/browse/FLINK-8910
> Project: Flink
>  Issue Type: Sub-task
>  Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>Priority: Major
> Fix For: 1.5.0
>
>
> We should have an automated end-to-end test that can run nightly to check 
> that sticky allocation and local recovery work as expected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5676: [FLINK-8910][tests] Automated end-to-end test for ...

[StefanRRichter]

Github user StefanRRichter commented on a diff in the pull request:

https://github.com/apache/flink/pull/5676#discussion_r182126796
  
--- Diff: 
flink-end-to-end-tests/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java
 ---
@@ -0,0 +1,451 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.tests;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.utils.ParameterTool;
+import org.apache.flink.contrib.streaming.state.RocksDBStateBackend;
+import org.apache.flink.runtime.state.CheckpointListener;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.runtime.state.filesystem.FsStateBackend;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.PrintSinkFunction;
+import org.apache.flink.streaming.api.functions.source.RichSourceFunction;
+import org.apache.flink.streaming.api.operators.StreamingRuntimeContext;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Automatic end-to-end test for local recovery (including sticky 
allocation).
+ */
+public class StickyAllocationAndLocalRecoveryTestJob {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(StickyAllocationAndLocalRecoveryTestJob.class);
+
+   public static void main(String[] args) throws Exception {
+
+   final ParameterTool pt = ParameterTool.fromArgs(args);
+
+   final StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
+
+   env.setParallelism(pt.getInt("parallelism", 1));
--- End diff --

👍 


---

[jira] [Assigned] (FLINK-8785) JobSubmitHandler does not handle JobSubmissionExceptions

[buptljy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

buptljy reassigned FLINK-8785:
--

Assignee: buptljy

> JobSubmitHandler does not handle JobSubmissionExceptions
> 
>
> Key: FLINK-8785
> URL: https://issues.apache.org/jira/browse/FLINK-8785
> Project: Flink
>  Issue Type: Bug
>  Components: Job-Submission, JobManager, REST
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: buptljy
>Priority: Critical
>  Labels: flip-6
>
> If the job submission, i.e. {{DispatcherGateway#submitJob}} fails with a 
> {{JobSubmissionException}} the {{JobSubmissionHandler}} returns "Internal 
> server error" instead of signaling the failed job submission.
> This can for example occur if the transmitted execution graph is faulty, as 
> tested by the \{{JobSubmissionFailsITCase}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8785) JobSubmitHandler does not handle JobSubmissionExceptions

[buptljy (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441047#comment-16441047
 ] 

buptljy commented on FLINK-8785:


[~Zentol] 

Okay, I've already known how to do this. I will assign this task to myself if 
you don't mind.

> JobSubmitHandler does not handle JobSubmissionExceptions
> 
>
> Key: FLINK-8785
> URL: https://issues.apache.org/jira/browse/FLINK-8785
> Project: Flink
>  Issue Type: Bug
>  Components: Job-Submission, JobManager, REST
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Priority: Critical
>  Labels: flip-6
>
> If the job submission, i.e. {{DispatcherGateway#submitJob}} fails with a 
> {{JobSubmissionException}} the {{JobSubmissionHandler}} returns "Internal 
> server error" instead of signaling the failed job submission.
> This can for example occur if the transmitted execution graph is faulty, as 
> tested by the \{{JobSubmissionFailsITCase}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5676: [FLINK-8910][tests] Automated end-to-end test for ...

[StefanRRichter]

Github user StefanRRichter commented on a diff in the pull request:

https://github.com/apache/flink/pull/5676#discussion_r182116943
  
--- Diff: flink-end-to-end-tests/test-scripts/common.sh ---
@@ -176,10 +176,40 @@ function s3_delete {
 https://${bucket}.s3.amazonaws.com/${s3_file}
 }
 
+function tm_watchdog {
+  expectedTm=$1
+  while true;
+  do
+runningTm=`jps | grep -o 'TaskManagerRunner' | wc -l`;
--- End diff --

👍 


---

[jira] [Commented] (FLINK-8910) Introduce automated end-to-end test for local recovery (including sticky scheduling)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441027#comment-16441027
 ] 

ASF GitHub Bot commented on FLINK-8910:
---

Github user StefanRRichter commented on a diff in the pull request:

https://github.com/apache/flink/pull/5676#discussion_r182117045
  
--- Diff: 
flink-end-to-end-tests/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java
 ---
@@ -0,0 +1,451 @@
+/**
--- End diff --

 


> Introduce automated end-to-end test for local recovery (including sticky 
> scheduling)
> 
>
> Key: FLINK-8910
> URL: https://issues.apache.org/jira/browse/FLINK-8910
> Project: Flink
>  Issue Type: Sub-task
>  Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>Priority: Major
> Fix For: 1.5.0
>
>
> We should have an automated end-to-end test that can run nightly to check 
> that sticky allocation and local recovery work as expected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8910) Introduce automated end-to-end test for local recovery (including sticky scheduling)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441026#comment-16441026
 ] 

ASF GitHub Bot commented on FLINK-8910:
---

Github user StefanRRichter commented on a diff in the pull request:

https://github.com/apache/flink/pull/5676#discussion_r182116943
  
--- Diff: flink-end-to-end-tests/test-scripts/common.sh ---
@@ -176,10 +176,40 @@ function s3_delete {
 https://${bucket}.s3.amazonaws.com/${s3_file}
 }
 
+function tm_watchdog {
+  expectedTm=$1
+  while true;
+  do
+runningTm=`jps | grep -o 'TaskManagerRunner' | wc -l`;
--- End diff --

 


> Introduce automated end-to-end test for local recovery (including sticky 
> scheduling)
> 
>
> Key: FLINK-8910
> URL: https://issues.apache.org/jira/browse/FLINK-8910
> Project: Flink
>  Issue Type: Sub-task
>  Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>Priority: Major
> Fix For: 1.5.0
>
>
> We should have an automated end-to-end test that can run nightly to check 
> that sticky allocation and local recovery work as expected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5676: [FLINK-8910][tests] Automated end-to-end test for ...

[StefanRRichter]

Github user StefanRRichter commented on a diff in the pull request:

https://github.com/apache/flink/pull/5676#discussion_r182117045
  
--- Diff: 
flink-end-to-end-tests/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java
 ---
@@ -0,0 +1,451 @@
+/**
--- End diff --

👍 


---

[jira] [Comment Edited] (FLINK-9193) Deprecate non-well-defined output methods on DataStream

[Timo Walther (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441023#comment-16441023
 ] 

Timo Walther edited comment on FLINK-9193 at 4/17/18 3:30 PM:
--

Do we really want to deprecate these useful methods? I think this will be a big 
barrier especially for new users. Of course they have no well-defined 
semantics, but are useful for debugging, prototyping, and testing. Not every 
user needs strong consistency guarantees either.


was (Author: twalthr):
Do we really want to deprecate these useful methods? I think this will a big 
barrier especially for new users. Of course they have no well-defined 
semantics, but are useful for debugging, prototyping, and testing. Not every 
user needs strong consistency guarantees either.

> Deprecate non-well-defined output methods on DataStream
> ---
>
> Key: FLINK-9193
> URL: https://issues.apache.org/jira/browse/FLINK-9193
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 1.5.0
>
>
> Some output methods on {{DataStream}} that write text to files are not safe 
> to use in a streaming program as they have no consistency guarantees. They 
> are:
>  - {{writeAsText()}}
>  - {{writeAsCsv()}}
>  - {{writeToSocket()}}
>  - {{writeUsingOutputFormat()}}
> Along with those we should also deprecate the {{SinkFunctions}} that they use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8910) Introduce automated end-to-end test for local recovery (including sticky scheduling)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441024#comment-16441024
 ] 

ASF GitHub Bot commented on FLINK-8910:
---

Github user StefanRRichter commented on a diff in the pull request:

https://github.com/apache/flink/pull/5676#discussion_r182116588
  
--- Diff: flink-end-to-end-tests/test-scripts/common.sh ---
@@ -176,10 +176,40 @@ function s3_delete {
 https://${bucket}.s3.amazonaws.com/${s3_file}
 }
 
+function tm_watchdog {
--- End diff --

 


> Introduce automated end-to-end test for local recovery (including sticky 
> scheduling)
> 
>
> Key: FLINK-8910
> URL: https://issues.apache.org/jira/browse/FLINK-8910
> Project: Flink
>  Issue Type: Sub-task
>  Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>Priority: Major
> Fix For: 1.5.0
>
>
> We should have an automated end-to-end test that can run nightly to check 
> that sticky allocation and local recovery work as expected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9193) Deprecate non-well-defined output methods on DataStream

[Timo Walther (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441023#comment-16441023
 ] 

Timo Walther commented on FLINK-9193:
-

Do we really want to deprecate these useful methods? I think this will a big 
barrier especially for new users. Of course they have no well-defined 
semantics, but are useful for debugging, prototyping, and testing. Not every 
user needs strong consistency guarantees either.

> Deprecate non-well-defined output methods on DataStream
> ---
>
> Key: FLINK-9193
> URL: https://issues.apache.org/jira/browse/FLINK-9193
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 1.5.0
>
>
> Some output methods on {{DataStream}} that write text to files are not safe 
> to use in a streaming program as they have no consistency guarantees. They 
> are:
>  - {{writeAsText()}}
>  - {{writeAsCsv()}}
>  - {{writeToSocket()}}
>  - {{writeUsingOutputFormat()}}
> Along with those we should also deprecate the {{SinkFunctions}} that they use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5676: [FLINK-8910][tests] Automated end-to-end test for ...

[StefanRRichter]

Github user StefanRRichter commented on a diff in the pull request:

https://github.com/apache/flink/pull/5676#discussion_r182116588
  
--- Diff: flink-end-to-end-tests/test-scripts/common.sh ---
@@ -176,10 +176,40 @@ function s3_delete {
 https://${bucket}.s3.amazonaws.com/${s3_file}
 }
 
+function tm_watchdog {
--- End diff --

👍 


---

[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441020#comment-16441020
 ] 

ASF GitHub Bot commented on FLINK-8703:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5665
  
I think this looks good to go then.  


> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5665: [FLINK-8703][tests] Port WebFrontendITCase to MiniCluster...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5665
  
I think this looks good to go then. 👍 


---

[GitHub] flink issue #5780: [FLINK-8955][tests] Port ClassLoaderITCase to flip6

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5780
  
still failures because of this?


---

[jira] [Commented] (FLINK-8955) Port ClassLoaderITCase to flip6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441019#comment-16441019
 ] 

ASF GitHub Bot commented on FLINK-8955:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5780
  
still failures because of this?


> Port ClassLoaderITCase to flip6
> ---
>
> Key: FLINK-8955
> URL: https://issues.apache.org/jira/browse/FLINK-8955
> Project: Flink
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7627) SingleElementIterable should implement with Serializable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440985#comment-16440985
 ] 

ASF GitHub Bot commented on FLINK-7627:
---

Github user hequn8128 closed the pull request at:

https://github.com/apache/flink/pull/4674


> SingleElementIterable should implement with Serializable 
> -
>
> Key: FLINK-7627
> URL: https://issues.apache.org/jira/browse/FLINK-7627
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
>
> {{SingleElementIterable}} is used to merge accumulators and it should be 
> serializable considering that it will be serialized when doing checkpoint.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9195) Delete non-well-defined output methods on DataStream

[Michael Latta (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440984#comment-16440984
 ] 

Michael Latta commented on FLINK-9195:
--

Those sinks are for debugging of simple jobs.  They are documented as such.  I 
see no need to force new users to create similar sinks for early debugging and 
familiarization efforts.  They were very useful in the initial stages of 
getting to know flink.

> Delete non-well-defined output methods on DataStream
> 
>
> Key: FLINK-9195
> URL: https://issues.apache.org/jira/browse/FLINK-9195
> Project: Flink
>  Issue Type: Improvement
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 1.6.0
>
>
> Some output methods on {{DataStream}} that write text to files are not safe 
> to use in a streaming program as they have no consistency guarantees. They 
> are:
>  - {{writeAsText()}}
>  - {{writeAsCsv()}}
>  - {{writeToSocket()}}
>  - {{writeUsingOutputFormat()}}
> Along with those we should also delete the {{SinkFunctions}} that they use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #4674: [FLINK-7627] [table] SingleElementIterable should ...

[hequn8128]

Github user hequn8128 closed the pull request at:

https://github.com/apache/flink/pull/4674


---

[jira] [Closed] (FLINK-7627) SingleElementIterable should implement with Serializable

[Hequn Cheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hequn Cheng closed FLINK-7627.
--
Resolution: Not A Problem

> SingleElementIterable should implement with Serializable 
> -
>
> Key: FLINK-7627
> URL: https://issues.apache.org/jira/browse/FLINK-7627
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
>
> {{SingleElementIterable}} is used to merge accumulators and it should be 
> serializable considering that it will be serialized when doing checkpoint.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9196) YARN: Flink binaries are not deleted from HDFS after cluster shutdown

[Gary Yao (JIRA)]

Gary Yao created FLINK-9196:
---

 Summary: YARN: Flink binaries are not deleted from HDFS after 
cluster shutdown
 Key: FLINK-9196
 URL: https://issues.apache.org/jira/browse/FLINK-9196
 Project: Flink
  Issue Type: Bug
  Components: YARN
Affects Versions: 1.5.0
Reporter: Gary Yao


When deploying on YARN in flip6 mode, the Flink binaries are not deleted from 
HDFS after the cluster shuts down.

*Steps to reproduce*
# Submit job in YARN job mode, non-detached:
{noformat} HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster 
-yjm 2048 -ytm 2048 -yn 2  ./examples/streaming/WordCount.jar {noformat}
# Check contents of {{/user/hadoop/.flink/}} on HDFS after job 
is finished:
{noformat}
[hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls 
/user/hadoop/.flink/application_1523966184826_0016
Found 6 items
-rw-r--r--   1 hadoop hadoop583 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml
-rw-r--r--   1 hadoop hadoop332 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp
-rw-r--r--   1 hadoop hadoop   89779342 2018-04-02 17:08 
/user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar
drwxrwxrwx   - hadoop hadoop  0 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/lib
-rw-r--r--   1 hadoop hadoop   1939 2018-04-02 15:37 
/user/hadoop/.flink/application_1523966184826_0016/log4j.properties
-rw-r--r--   1 hadoop hadoop   2331 2018-04-02 15:37 
/user/hadoop/.flink/application_1523966184826_0016/logback.xml
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7627) SingleElementIterable should implement with Serializable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440979#comment-16440979
 ] 

ASF GitHub Bot commented on FLINK-7627:
---

Github user hequn8128 commented on the issue:

https://github.com/apache/flink/pull/4674
  
@fhueske Hi, I will close it. Thanks for the checking. 


> SingleElementIterable should implement with Serializable 
> -
>
> Key: FLINK-7627
> URL: https://issues.apache.org/jira/browse/FLINK-7627
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
>
> {{SingleElementIterable}} is used to merge accumulators and it should be 
> serializable considering that it will be serialized when doing checkpoint.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #4674: [FLINK-7627] [table] SingleElementIterable should impleme...

[hequn8128]

Github user hequn8128 commented on the issue:

https://github.com/apache/flink/pull/4674
  
@fhueske Hi, I will close it. Thanks for the checking. 


---

[jira] [Commented] (FLINK-8366) Use Row instead of String as key when process upsert results

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440978#comment-16440978
 ] 

ASF GitHub Bot commented on FLINK-8366:
---

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/5244
  
merging


> Use Row instead of String as key when process upsert results
> 
>
> Key: FLINK-8366
> URL: https://issues.apache.org/jira/browse/FLINK-8366
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
>
> In {{TableSinkITCase.upsertResults()}} function, we use String as key to 
> upsert results. This will make (1,11) and (11,1) have the same key (i.e., 
> 111).
> This bugfix use Row instead of String to avoid the String problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5244: [FLINK-8366] [table] Use Row instead of String as key whe...

[fhueske]

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/5244
  
merging


---

[jira] [Commented] (FLINK-6924) ADD LOG(X) supported in TableAPI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440975#comment-16440975
 ] 

ASF GitHub Bot commented on FLINK-6924:
---

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/5638
  
Thanks for the PR @buptljy!

As mentioned in the comment, the function should be exposed similar to 
`ln()` and `log10()` as `expr.log(base)`. For that the expression parser would 
need to be adapted.

I'll do that and also update docs and tests to reflect these changes before 
merging.


> ADD LOG(X) supported in TableAPI
> 
>
> Key: FLINK-6924
> URL: https://issues.apache.org/jira/browse/FLINK-6924
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: buptljy
>Priority: Major
>  Labels: starter
>
> See FLINK-6891 for detail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5638: [FLINK-6924][table]ADD LOG(X) supported in TableAPI

[fhueske]

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/5638
  
Thanks for the PR @buptljy!

As mentioned in the comment, the function should be exposed similar to 
`ln()` and `log10()` as `expr.log(base)`. For that the expression parser would 
need to be adapted.

I'll do that and also update docs and tests to reflect these changes before 
merging.


---

[jira] [Commented] (FLINK-6924) ADD LOG(X) supported in TableAPI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440971#comment-16440971
 ] 

ASF GitHub Bot commented on FLINK-6924:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/5638#discussion_r182104673
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/scala/expressionDsl.scala
 ---
@@ -1130,4 +1130,13 @@ object concat_ws {
   }
 }
 
+object log {
+  def apply(base: Expression, antilogarithm: Expression): Expression = {
+Log(base, antilogarithm)
+  }
+  def apply(antilogarithm: Expression): Expression = {
+new Log(antilogarithm)
--- End diff --

We should go for `antilog.log(base)` for consistency with `antilog.ln()` 
and `antilog.log10()`. 
Since we need to inverse the parameters of the case class, we need to adapt 
the `ExpressionParser`.


> ADD LOG(X) supported in TableAPI
> 
>
> Key: FLINK-6924
> URL: https://issues.apache.org/jira/browse/FLINK-6924
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: buptljy
>Priority: Major
>  Labels: starter
>
> See FLINK-6891 for detail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5638: [FLINK-6924][table]ADD LOG(X) supported in TableAP...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/5638#discussion_r182104673
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/scala/expressionDsl.scala
 ---
@@ -1130,4 +1130,13 @@ object concat_ws {
   }
 }
 
+object log {
+  def apply(base: Expression, antilogarithm: Expression): Expression = {
+Log(base, antilogarithm)
+  }
+  def apply(antilogarithm: Expression): Expression = {
+new Log(antilogarithm)
--- End diff --

We should go for `antilog.log(base)` for consistency with `antilog.ln()` 
and `antilog.log10()`. 
Since we need to inverse the parameters of the case class, we need to adapt 
the `ExpressionParser`.


---

[jira] [Commented] (FLINK-8990) End-to-end test: Dynamic Kafka partition discovery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440959#comment-16440959
 ] 

ASF GitHub Bot commented on FLINK-8990:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5779
  
The changes look good to merge!  


> End-to-end test: Dynamic Kafka partition discovery
> --
>
> Key: FLINK-8990
> URL: https://issues.apache.org/jira/browse/FLINK-8990
> Project: Flink
>  Issue Type: Sub-task
>  Components: Kafka Connector, Tests
>Reporter: Till Rohrmann
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
>
> We should add an end-to-end test which verifies the dynamic partition 
> discovery of Flink's Kafka connector. We can simulate it by reading from a 
> Kafka topic to which we add partitions after the job started. By writing to 
> these new partitions it should be verifiable whether Flink noticed them by 
> checking the output for completeness.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5779: [FLINK-8990] [test] Extend Kafka end-to-end test to verif...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5779
  
The changes look good to merge! 👍 


---

[jira] [Created] (FLINK-9195) Delete non-well-defined output methods on DataStream

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-9195:
---

 Summary: Delete non-well-defined output methods on DataStream
 Key: FLINK-9195
 URL: https://issues.apache.org/jira/browse/FLINK-9195
 Project: Flink
  Issue Type: Improvement
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
 Fix For: 1.6.0


Some output methods on {{DataStream}} that write text to files are not safe to 
use in a streaming program as they have no consistency guarantees. They are:
 - {{writeAsText()}}
 - {{writeAsCsv()}}
 - {{writeToSocket()}}
 - {{writeUsingOutputFormat()}}

Along with those we should also delete the {{SinkFunctions}} that they use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9194) Finished jobs are not archived to HistoryServer

[Gary Yao (JIRA)]

Gary Yao created FLINK-9194:
---

 Summary: Finished jobs are not archived to HistoryServer
 Key: FLINK-9194
 URL: https://issues.apache.org/jira/browse/FLINK-9194
 Project: Flink
  Issue Type: Bug
  Components: History Server, JobManager
Affects Versions: 1.5.0
 Environment: Flink 2af481a
Reporter: Gary Yao


In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9192) Undo parameterization of StateMachine Example

[Stefan Richter (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440934#comment-16440934
 ] 

Stefan Richter commented on FLINK-9192:
---

+1, I think that the test could build upon the universal test from FLINK-8971 
which can do the same validation and already offers all the configuration 
parameters.

> Undo parameterization of StateMachine Example
> -
>
> Key: FLINK-9192
> URL: https://issues.apache.org/jira/browse/FLINK-9192
> Project: Flink
>  Issue Type: Improvement
>Reporter: Stephan Ewen
>Priority: Major
>
> The example has been changed to add parametrization and a different sink.
> I would vote to undo these changes, the make the example less nice and use 
> non-recommended sinks:
>   - For state backend, incremental checkpoints, async checkpoints, etc. 
> having these settings in the example blows up the parameter list of the 
> example and distracts from what the example is about.
>   - If the main reason for this is an end-to-end test, then these settings 
> should go into the test's Flink config.
>   - The {{writeAsText}} is a sink that is not recommended to use, because it 
> is not integrated with checkpoints and has no well defined semantics.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9192) Undo parameterization of StateMachine Example

[Kostas Kloudas (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440933#comment-16440933
 ] 

Kostas Kloudas commented on FLINK-9192:
---

These changes were made in the context of the end-to-end test for High 
Availability.

It was an intermediate step until the general purpose testing job is written, 
which will replace the StateMachine job in the test.

That said, I am totally up for undoing the changes.

> Undo parameterization of StateMachine Example
> -
>
> Key: FLINK-9192
> URL: https://issues.apache.org/jira/browse/FLINK-9192
> Project: Flink
>  Issue Type: Improvement
>Reporter: Stephan Ewen
>Priority: Major
>
> The example has been changed to add parametrization and a different sink.
> I would vote to undo these changes, the make the example less nice and use 
> non-recommended sinks:
>   - For state backend, incremental checkpoints, async checkpoints, etc. 
> having these settings in the example blows up the parameter list of the 
> example and distracts from what the example is about.
>   - If the main reason for this is an end-to-end test, then these settings 
> should go into the test's Flink config.
>   - The {{writeAsText}} is a sink that is not recommended to use, because it 
> is not integrated with checkpoints and has no well defined semantics.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (FLINK-8900) YARN FinalStatus always shows as KILLED with Flip-6

[Gary Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440927#comment-16440927
 ] 

Gary Yao edited comment on FLINK-8900 at 4/17/18 2:37 PM:
--

When submitting in non-detached mode, the problem still surfaces. It detached 
mode the status is set correctly.

Command used to submit:
{noformat}
HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster -yjm 2048 
-ytm 2048  ./examples/streaming/WordCount.jar
{noformat}

State and FinalStatus is: KILLED

I re-opened the ticket.




was (Author: gjy):
When submitting in non-detached mode, the problem still surfaces. It detached 
mode the status is set correctly.

Command used to submit:
{noformat}
HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster -yjm 2048 
-ytm 2048  ./examples/streaming/WordCount.jar
{noformat}

State and FinalStatus is: KILLED



> YARN FinalStatus always shows as KILLED with Flip-6
> ---
>
> Key: FLINK-8900
> URL: https://issues.apache.org/jira/browse/FLINK-8900
> Project: Flink
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Nico Kruber
>Assignee: Till Rohrmann
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Whenever I run a simple simple word count like this one on YARN with Flip-6 
> enabled,
> {code}
> ./bin/flink run -m yarn-cluster -yjm 768 -ytm 3072 -ys 2 -p 20 -c 
> org.apache.flink.streaming.examples.wordcount.WordCount 
> ./examples/streaming/WordCount.jar --input /usr/share/doc/rsync-3.0.6/COPYING
> {code}
> it will show up as {{KILLED}} in the {{State}} and {{FinalStatus}} columns 
> even though the program ran successfully like this one (irrespective of 
> FLINK-8899 occurring or not):
> {code}
> 2018-03-08 16:48:39,049 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job Streaming 
> WordCount (11a794d2f5dc2955d8015625ec300c20) switched from state RUNNING to 
> FINISHED.
> 2018-03-08 16:48:39,050 INFO  
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping 
> checkpoint coordinator for job 11a794d2f5dc2955d8015625ec300c20
> 2018-03-08 16:48:39,050 INFO  
> org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore  - 
> Shutting down
> 2018-03-08 16:48:39,078 INFO  
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher  - Job 
> 11a794d2f5dc2955d8015625ec300c20 reached globally terminal state FINISHED.
> 2018-03-08 16:48:39,151 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Register 
> TaskManager e58efd886429e8f080815ea74ddfa734 at the SlotManager.
> 2018-03-08 16:48:39,221 INFO  org.apache.flink.runtime.jobmaster.JobMaster
>   - Stopping the JobMaster for job Streaming 
> WordCount(11a794d2f5dc2955d8015625ec300c20).
> 2018-03-08 16:48:39,270 INFO  org.apache.flink.runtime.jobmaster.JobMaster
>   - Close ResourceManager connection 
> 43f725adaee14987d3ff99380701f52f: JobManager is shutting down..
> 2018-03-08 16:48:39,270 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - Disconnect job manager 
> 0...@akka.tcp://fl...@ip-172-31-7-0.eu-west-1.compute.internal:34281/user/jobmanager_0
>  for job 11a794d2f5dc2955d8015625ec300c20 from the resource manager.
> 2018-03-08 16:48:39,349 INFO  
> org.apache.flink.runtime.jobmaster.slotpool.SlotPool  - Suspending 
> SlotPool.
> 2018-03-08 16:48:39,349 INFO  
> org.apache.flink.runtime.jobmaster.slotpool.SlotPool  - Stopping 
> SlotPool.
> 2018-03-08 16:48:39,349 INFO  
> org.apache.flink.runtime.jobmaster.JobManagerRunner   - 
> JobManagerRunner already shutdown.
> 2018-03-08 16:48:39,775 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Register 
> TaskManager 4e1fb6c8f95685e24b6a4cb4b71ffb92 at the SlotManager.
> 2018-03-08 16:48:39,846 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Register 
> TaskManager b5bce0bdfa7fbb0f4a0905cc3ee1c233 at the SlotManager.
> 2018-03-08 16:48:39,876 INFO  
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - RECEIVED 
> SIGNAL 15: SIGTERM. Shutting down as requested.
> 2018-03-08 16:48:39,910 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Register 
> TaskManager a35b0690fdc6ec38bbcbe18a965000fd at the SlotManager.
> 2018-03-08 16:48:39,942 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Register 
> TaskManager 5175cabe428bea19230ac056ff2a17bb at the SlotManager.
> 2018-03-08 16:48:39,974 INFO  org.apache.flink.runtime.blob.BlobServer
>   - Stopped BLOB server at 0.0.0.0:46511
> 2018-03-08 16:48:39,975 INFO  
> 

[jira] [Reopened] (FLINK-8900) YARN FinalStatus always shows as KILLED with Flip-6

[Gary Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao reopened FLINK-8900:
-

When submitting in non-detached mode, the problem still surfaces. It detached 
mode the status is set correctly.

Command used to submit:
{noformat}
HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster -yjm 2048 
-ytm 2048  ./examples/streaming/WordCount.jar
{noformat}

State and FinalStatus is: KILLED



> YARN FinalStatus always shows as KILLED with Flip-6
> ---
>
> Key: FLINK-8900
> URL: https://issues.apache.org/jira/browse/FLINK-8900
> Project: Flink
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Nico Kruber
>Assignee: Till Rohrmann
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Whenever I run a simple simple word count like this one on YARN with Flip-6 
> enabled,
> {code}
> ./bin/flink run -m yarn-cluster -yjm 768 -ytm 3072 -ys 2 -p 20 -c 
> org.apache.flink.streaming.examples.wordcount.WordCount 
> ./examples/streaming/WordCount.jar --input /usr/share/doc/rsync-3.0.6/COPYING
> {code}
> it will show up as {{KILLED}} in the {{State}} and {{FinalStatus}} columns 
> even though the program ran successfully like this one (irrespective of 
> FLINK-8899 occurring or not):
> {code}
> 2018-03-08 16:48:39,049 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job Streaming 
> WordCount (11a794d2f5dc2955d8015625ec300c20) switched from state RUNNING to 
> FINISHED.
> 2018-03-08 16:48:39,050 INFO  
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping 
> checkpoint coordinator for job 11a794d2f5dc2955d8015625ec300c20
> 2018-03-08 16:48:39,050 INFO  
> org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore  - 
> Shutting down
> 2018-03-08 16:48:39,078 INFO  
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher  - Job 
> 11a794d2f5dc2955d8015625ec300c20 reached globally terminal state FINISHED.
> 2018-03-08 16:48:39,151 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Register 
> TaskManager e58efd886429e8f080815ea74ddfa734 at the SlotManager.
> 2018-03-08 16:48:39,221 INFO  org.apache.flink.runtime.jobmaster.JobMaster
>   - Stopping the JobMaster for job Streaming 
> WordCount(11a794d2f5dc2955d8015625ec300c20).
> 2018-03-08 16:48:39,270 INFO  org.apache.flink.runtime.jobmaster.JobMaster
>   - Close ResourceManager connection 
> 43f725adaee14987d3ff99380701f52f: JobManager is shutting down..
> 2018-03-08 16:48:39,270 INFO  org.apache.flink.yarn.YarnResourceManager   
>   - Disconnect job manager 
> 0...@akka.tcp://fl...@ip-172-31-7-0.eu-west-1.compute.internal:34281/user/jobmanager_0
>  for job 11a794d2f5dc2955d8015625ec300c20 from the resource manager.
> 2018-03-08 16:48:39,349 INFO  
> org.apache.flink.runtime.jobmaster.slotpool.SlotPool  - Suspending 
> SlotPool.
> 2018-03-08 16:48:39,349 INFO  
> org.apache.flink.runtime.jobmaster.slotpool.SlotPool  - Stopping 
> SlotPool.
> 2018-03-08 16:48:39,349 INFO  
> org.apache.flink.runtime.jobmaster.JobManagerRunner   - 
> JobManagerRunner already shutdown.
> 2018-03-08 16:48:39,775 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Register 
> TaskManager 4e1fb6c8f95685e24b6a4cb4b71ffb92 at the SlotManager.
> 2018-03-08 16:48:39,846 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Register 
> TaskManager b5bce0bdfa7fbb0f4a0905cc3ee1c233 at the SlotManager.
> 2018-03-08 16:48:39,876 INFO  
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - RECEIVED 
> SIGNAL 15: SIGTERM. Shutting down as requested.
> 2018-03-08 16:48:39,910 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Register 
> TaskManager a35b0690fdc6ec38bbcbe18a965000fd at the SlotManager.
> 2018-03-08 16:48:39,942 INFO  
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Register 
> TaskManager 5175cabe428bea19230ac056ff2a17bb at the SlotManager.
> 2018-03-08 16:48:39,974 INFO  org.apache.flink.runtime.blob.BlobServer
>   - Stopped BLOB server at 0.0.0.0:46511
> 2018-03-08 16:48:39,975 INFO  
> org.apache.flink.runtime.blob.TransientBlobCache  - Shutting down 
> BLOB cache
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9193) Deprecate non-well-defined output methods on DataStream

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-9193:
---

 Summary: Deprecate non-well-defined output methods on DataStream
 Key: FLINK-9193
 URL: https://issues.apache.org/jira/browse/FLINK-9193
 Project: Flink
  Issue Type: Improvement
  Components: DataStream API
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
 Fix For: 1.5.0


Some output methods on {{DataStream}} that write text to files are not safe to 
use in a streaming program as they have no consistency guarantees. They are:
 - {{writeAsText()}}
 - {{writeAsCsv()}}
 - {{writeToSocket()}}
 - {{writeUsingOutputFormat()}}

Along with those we should also deprecate the {{SinkFunctions}} that they use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9192) Undo parameterization of StateMachine Example

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-9192:
---

 Summary: Undo parameterization of StateMachine Example
 Key: FLINK-9192
 URL: https://issues.apache.org/jira/browse/FLINK-9192
 Project: Flink
  Issue Type: Improvement
Reporter: Stephan Ewen


The example has been changed to add parametrization and a different sink.

I would vote to undo these changes, the make the example less nice and use 
non-recommended sinks:

  - For state backend, incremental checkpoints, async checkpoints, etc. having 
these settings in the example blows up the parameter list of the example and 
distracts from what the example is about.
  - If the main reason for this is an end-to-end test, then these settings 
should go into the test's Flink config.
  - The {{writeAsText}} is a sink that is not recommended to use, because it is 
not integrated with checkpoints and has no well defined semantics.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)