date:20180218

[jira] [Assigned] (FLINK-6615) tmp directory not cleaned up on shutdown

2018-02-18 Thread Bowen Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-6615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bowen Li reassigned FLINK-6615:
---

Assignee: (was: Bowen Li)

> tmp directory not cleaned up on shutdown
> 
>
> Key: FLINK-6615
> URL: https://issues.apache.org/jira/browse/FLINK-6615
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.2.0
>Reporter: Andrey
>Priority: Major
>
> Steps to reproduce:
> 1) Stop task manager gracefully (kill -6 )
> 2) In the logs:
> {code}
> 2017-05-17 13:35:50,147 INFO  org.apache.zookeeper.ClientCnxn 
>   - EventThread shut down [main-EventThread]
> 2017-05-17 13:35:50,200 ERROR 
> org.apache.flink.runtime.io.disk.iomanager.IOManager  - IOManager 
> failed to properly clean up temp file directory: 
> /mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47 
> [flink-akka.actor.default-dispatcher-2]
> java.nio.file.DirectoryNotEmptyException: 
> /mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47
>   at 
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
>   at 
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
>   at java.nio.file.Files.delete(Files.java:1126)
>   at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:154)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.IOManager.shutdown(IOManager.java:109)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.shutdown(IOManagerAsync.java:185)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager.postStop(TaskManager.scala:241)
>   at akka.actor.Actor$class.aroundPostStop(Actor.scala:477)
> {code}
> Expected:
> * on shutdown delete non-empty directory anyway. 
> Notes:
> * after process terminated, I've checked 
> "/mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47" directory and 
> didn't find anything there. So it looks like timing issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-6615) tmp directory not cleaned up on shutdown

2018-02-18 Thread Bowen Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-6615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bowen Li resolved FLINK-6615.
-
Resolution: Won't Fix

> tmp directory not cleaned up on shutdown
> 
>
> Key: FLINK-6615
> URL: https://issues.apache.org/jira/browse/FLINK-6615
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.2.0
>Reporter: Andrey
>Assignee: Bowen Li
>Priority: Major
>
> Steps to reproduce:
> 1) Stop task manager gracefully (kill -6 )
> 2) In the logs:
> {code}
> 2017-05-17 13:35:50,147 INFO  org.apache.zookeeper.ClientCnxn 
>   - EventThread shut down [main-EventThread]
> 2017-05-17 13:35:50,200 ERROR 
> org.apache.flink.runtime.io.disk.iomanager.IOManager  - IOManager 
> failed to properly clean up temp file directory: 
> /mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47 
> [flink-akka.actor.default-dispatcher-2]
> java.nio.file.DirectoryNotEmptyException: 
> /mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47
>   at 
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
>   at 
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
>   at java.nio.file.Files.delete(Files.java:1126)
>   at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:154)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.IOManager.shutdown(IOManager.java:109)
>   at 
> org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.shutdown(IOManagerAsync.java:185)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager.postStop(TaskManager.scala:241)
>   at akka.actor.Actor$class.aroundPostStop(Actor.scala:477)
> {code}
> Expected:
> * on shutdown delete non-empty directory anyway. 
> Notes:
> * after process terminated, I've checked 
> "/mnt/flink/tmp/flink-io-66f1d0ec-8976-41bf-9575-f80b181b0e47" directory and 
> didn't find anything there. So it looks like timing issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-7947) Let ParameterTool return a dedicated GlobalJobParameters object

2018-02-18 Thread Bowen Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-7947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bowen Li reassigned FLINK-7947:
---

Assignee: (was: Bowen Li)

> Let ParameterTool return a dedicated GlobalJobParameters object
> ---
>
> Key: FLINK-7947
> URL: https://issues.apache.org/jira/browse/FLINK-7947
> Project: Flink
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.4.0
>Reporter: Till Rohrmann
>Priority: Major
>
> The {{ParameterTool}} directly implements the {{GlobalJobParameters}} 
> interface. Additionally it has grown over time to not only store the 
> configuration parameters but also to record which parameters have been 
> requested and what default value was set. This information is irrelevant on 
> the server side when setting a {{GlobalJobParameters}} object via 
> {{ExecutionConfig#setGlobalJobParameters}}.
> Since we don't separate the {{ParameterTool}} logic and the actual data view, 
> users ran into problems when reusing the same {{ParameterTool}} to start 
> multiple jobs concurrently (see FLINK-7943). I think it would be a much 
> clearer separation of concerns if we would actually split the 
> {{GlobalJobParameters}} from the {{ParameterTool}}.
> Furthermore, we should think about whether {{ParameterTool#get}} should have 
> side effects or not as it does right now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #:

2018-02-18 Thread jimmec

Github user jimmec commented on the pull request:


https://github.com/apache/flink/commit/3062e29a8bbb667d098c43a0b225d5602049#commitcomment-27642949
  
In 
flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcherTest.java:
In 
flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcherTest.java
 on line 113:
Is DummyFlink`Kafka`Consumer a typo?? It's at the least inconsistent with 
the comment above this line.


---

[jira] [Closed] (FLINK-8548) Add Streaming State Machine Example

2018-02-18 Thread Stephan Ewen (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-8548.
---

> Add Streaming State Machine Example
> ---
>
> Key: FLINK-8548
> URL: https://issues.apache.org/jira/browse/FLINK-8548
> Project: Flink
>  Issue Type: Sub-task
>  Components: Examples
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Major
> Fix For: 1.5.0
>
>
> Add the example from 
> https://github.com/StephanEwen/flink-demos/tree/master/streaming-state-machine
>  to the Flink examples.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-8548) Add Streaming State Machine Example

2018-02-18 Thread Stephan Ewen (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-8548.
-
Resolution: Done

Done in 80883feeeba5232ecd41fe6759c69bf2ff1a6483

> Add Streaming State Machine Example
> ---
>
> Key: FLINK-8548
> URL: https://issues.apache.org/jira/browse/FLINK-8548
> Project: Flink
>  Issue Type: Sub-task
>  Components: Examples
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Major
> Fix For: 1.5.0
>
>
> Add the example from 
> https://github.com/StephanEwen/flink-demos/tree/master/streaming-state-machine
>  to the Flink examples.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8680) Name printing sinks by default.

2018-02-18 Thread Stephan Ewen (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-8680.
---

> Name printing sinks by default.
> ---
>
> Key: FLINK-8680
> URL: https://issues.apache.org/jira/browse/FLINK-8680
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Trivial
> Fix For: 1.5.0
>
>
> The sinks that pring to std. out and std. err show up as "Sink: Unnamed" in 
> logs and the UI.
> They should be named "Print to Std. Out" and "Print to Std. Err" by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-8549) Move TimerServiceOptions to TaskManagerOptions

2018-02-18 Thread Stephan Ewen (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-8549.
-
Resolution: Fixed

Fixed via 85bfc073680667e9daa16439d33fa86c1c0be133

> Move TimerServiceOptions to TaskManagerOptions
> --
>
> Key: FLINK-8549
> URL: https://issues.apache.org/jira/browse/FLINK-8549
> Project: Flink
>  Issue Type: Improvement
>  Components: Local Runtime
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Major
> Fix For: 1.5.0
>
>
> The {{TimerServiceOptions}} are in the wrong place (prohibit generation of 
> config docs) and cause over-fragmentation of the options in the code base.
> I propose to simple move the one option from that class to the 
> {{TaskManagerOptions}}, as it relates to task execution. Other shutdown 
> related options are in there already.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-8680) Name printing sinks by default.

2018-02-18 Thread Stephan Ewen (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-8680.
-
Resolution: Fixed

Fixed via 21491b2717e3a60e63dcf4fb0aa839b9885be67a

> Name printing sinks by default.
> ---
>
> Key: FLINK-8680
> URL: https://issues.apache.org/jira/browse/FLINK-8680
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Trivial
> Fix For: 1.5.0
>
>
> The sinks that pring to std. out and std. err show up as "Sink: Unnamed" in 
> logs and the UI.
> They should be named "Print to Std. Out" and "Print to Std. Err" by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8549) Move TimerServiceOptions to TaskManagerOptions

2018-02-18 Thread Stephan Ewen (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-8549.
---

> Move TimerServiceOptions to TaskManagerOptions
> --
>
> Key: FLINK-8549
> URL: https://issues.apache.org/jira/browse/FLINK-8549
> Project: Flink
>  Issue Type: Improvement
>  Components: Local Runtime
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Major
> Fix For: 1.5.0
>
>
> The {{TimerServiceOptions}} are in the wrong place (prohibit generation of 
> config docs) and cause over-fragmentation of the options in the code base.
> I propose to simple move the one option from that class to the 
> {{TaskManagerOptions}}, as it relates to task execution. Other shutdown 
> related options are in there already.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8548) Add Streaming State Machine Example

2018-02-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368705#comment-16368705
 ] 

ASF GitHub Bot commented on FLINK-8548:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5401


> Add Streaming State Machine Example
> ---
>
> Key: FLINK-8548
> URL: https://issues.apache.org/jira/browse/FLINK-8548
> Project: Flink
>  Issue Type: Sub-task
>  Components: Examples
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Major
> Fix For: 1.5.0
>
>
> Add the example from 
> https://github.com/StephanEwen/flink-demos/tree/master/streaming-state-machine
>  to the Flink examples.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8682) Make start/stop cluster scripts work without SSH for local HA setups

2018-02-18 Thread Stephan Ewen (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-8682.
---

> Make start/stop cluster scripts work without SSH for local HA setups
> 
>
> Key: FLINK-8682
> URL: https://issues.apache.org/jira/browse/FLINK-8682
> Project: Flink
>  Issue Type: Improvement
>  Components: Startup Shell Scripts
>Affects Versions: 1.4.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Minor
> Fix For: 1.5.0
>
>
> The startup should work for purely local (testing) cluster without setups 
> without SSH.
> While the shell scripts handle this correctly for TaskManagers, they don't 
> handle it correctly for JobManagers. As a consequence, {{start-cluster.sh}} 
> does not work without SSH when high availability is enabled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-8682) Make start/stop cluster scripts work without SSH for local HA setups

2018-02-18 Thread Stephan Ewen (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-8682.
-
Resolution: Fixed

Fixed via 39711fb6f1d801e067e121803ea82b1c2742c956

> Make start/stop cluster scripts work without SSH for local HA setups
> 
>
> Key: FLINK-8682
> URL: https://issues.apache.org/jira/browse/FLINK-8682
> Project: Flink
>  Issue Type: Improvement
>  Components: Startup Shell Scripts
>Affects Versions: 1.4.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Minor
> Fix For: 1.5.0
>
>
> The startup should work for purely local (testing) cluster without setups 
> without SSH.
> While the shell scripts handle this correctly for TaskManagers, they don't 
> handle it correctly for JobManagers. As a consequence, {{start-cluster.sh}} 
> does not work without SSH when high availability is enabled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5402: [FLINK-8549] [config] Move TimerServiceOptions int...

2018-02-18 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5402


---

[jira] [Commented] (FLINK-8549) Move TimerServiceOptions to TaskManagerOptions

2018-02-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368704#comment-16368704
 ] 

ASF GitHub Bot commented on FLINK-8549:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5402


> Move TimerServiceOptions to TaskManagerOptions
> --
>
> Key: FLINK-8549
> URL: https://issues.apache.org/jira/browse/FLINK-8549
> Project: Flink
>  Issue Type: Improvement
>  Components: Local Runtime
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Major
> Fix For: 1.5.0
>
>
> The {{TimerServiceOptions}} are in the wrong place (prohibit generation of 
> config docs) and cause over-fragmentation of the options in the code base.
> I propose to simple move the one option from that class to the 
> {{TaskManagerOptions}}, as it relates to task execution. Other shutdown 
> related options are in there already.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8681) Remove planVisualizer.html move notice

2018-02-18 Thread Stephan Ewen (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-8681.
---

> Remove planVisualizer.html move notice
> --
>
> Key: FLINK-8681
> URL: https://issues.apache.org/jira/browse/FLINK-8681
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Affects Versions: 1.4.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Trivial
> Fix For: 1.5.0
>
>
> The {{planVisualizer.html}} for optimizer plans is no longer in the Flink 
> distribution, but we hold a notice there that the visualizer has moved to the 
> website.
> That notice has been there for many versions (since Flink 1.0) and can be 
> removed now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-8681) Remove planVisualizer.html move notice

2018-02-18 Thread Stephan Ewen (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-8681.
-
Resolution: Fixed

Fixed via 096bf80d93d4f27b2e57aaff8754ebbe54151ecb

> Remove planVisualizer.html move notice
> --
>
> Key: FLINK-8681
> URL: https://issues.apache.org/jira/browse/FLINK-8681
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Affects Versions: 1.4.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Trivial
> Fix For: 1.5.0
>
>
> The {{planVisualizer.html}} for optimizer plans is no longer in the Flink 
> distribution, but we hold a notice there that the visualizer has moved to the 
> website.
> That notice has been there for many versions (since Flink 1.0) and can be 
> removed now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5401: [FLINK-8548] [examples] Add state machine example

2018-02-18 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5401


---

[jira] [Commented] (FLINK-8599) Improve the failure behavior of the FileInputFormat for bad files

2018-02-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368678#comment-16368678
 ] 

ASF GitHub Bot commented on FLINK-8599:
---

GitHub user ChengzhiZhao opened a pull request:

https://github.com/apache/flink/pull/5521

[FLINK-8599] Improve the failure behavior of the FileInputFormat for …

## What is the purpose of the change

This pull request is intent to improve the failure behavior of the 
ContinuousFileReader, currently if a bad file (for example, a different schema 
been dropped in this folder) came to the path and flink will do several retries.
However, since the file path persist in the checkpoint, when people tried 
to resume from external checkpoint, it threw the following error on no file 
been found and the process cannot move forward.

`java.io.IOException: Error opening the Input Split s3a://myfile [0,904]: 
No such file or directory: s3a://myfile`

The change is to check if the path exist before open the file, if error 
occurs and bad file removed, flink should resume the process and continue.

## Brief change log
- *Add a file exist check before open the file *

## Verifying this change
- *Manually verified the change by introduce a bad file while continuously 
monitoring the folder, after remove the bad file, the process continued.*

## Does this pull request potentially affect one of the following parts:
  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ChengzhiZhao/flink 
Improve_failure_behavior_FileInputFormat

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5521.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5521


commit 6fa8ef212c536acee56b2e9831ec92d1059449ff
Author: Chengzhi Zhao 
Date:   2018-02-18T18:23:32Z

[FLINK-8599] Improve the failure behavior of the FileInputFormat for bad 
files




> Improve the failure behavior of the FileInputFormat for bad files
> -
>
> Key: FLINK-8599
> URL: https://issues.apache.org/jira/browse/FLINK-8599
> Project: Flink
>  Issue Type: New Feature
>  Components: DataStream API
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Chengzhi Zhao
>Priority: Major
>
> So we have a s3 path that flink is monitoring that path to see new files 
> available.
> {code:java}
> val avroInputStream_activity = env.readFile(format, path, 
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1)  {code}
>  
> I am doing both internal and external check pointing and let's say there is a 
> bad file (for example, a different schema been dropped in this folder) came 
> to the path and flink will do several retries. I want to take those bad files 
> and let the process continue. However, since the file path persist in the 
> checkpoint, when I try to resume from external checkpoint, it threw the 
> following error on no file been found.
>  
> {code:java}
> java.io.IOException: Error opening the Input Split s3a://myfile [0,904]: No 
> such file or directory: s3a://myfile{code}
>  
> As [~fhue...@gmail.com] suggested, we could check if a path exists and before 
> trying to read a file and ignore the input split instead of throwing an 
> exception and causing a failure.
>  
> Also, I am thinking about add an error output for bad files as an option to 
> users. So if there is any bad files exist we could move them in a separated 
> path and do further analysis. 
>  
> Not sure how people feel about it, but I'd like to contribute on it if people 
> think this can be an improvement. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-7477) Use "hadoop classpath" to augment classpath when available

2018-02-18 Thread Ken Krugler (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368677#comment-16368677
 ] 

Ken Krugler commented on FLINK-7477:


It works (at least when running with YARN via EMR). I believe that's because 
the version of Hadoop on the EMR master matches what we're running against; on 
my machine, I have to switch between multiple versions of Hadoop for various 
(consulting) clients who are on different versions of Hadoop, and my {{hadoop}} 
symlink wound up pointing to a different version of Hadoop than what Flink was 
using.

Related note - the 1.4 release fixed some shading issues we were running into 
with AWS client classes (mostly around {{HttpCore}} stuff), but to get 
everything working properly I felt like I did some voodoo with class exclusions 
in the {{maven-shade-plugin}} section of my {{pom.xml}}, which still feels 
fragile.

> Use "hadoop classpath" to augment classpath when available
> --
>
> Key: FLINK-7477
> URL: https://issues.apache.org/jira/browse/FLINK-7477
> Project: Flink
>  Issue Type: Bug
>  Components: Startup Shell Scripts
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 1.4.0
>
>
> Currently, some cloud environments don't properly put the Hadoop jars into 
> {{HADOOP_CLASSPATH}} (or don't set {{HADOOP_CLASSPATH}}) at all. We should 
> check in {{config.sh}} if the {{hadoop}} binary is on the path and augment 
> our {{INTERNAL_HADOOP_CLASSPATHS}} with the result of {{hadoop classpath}} in 
> our scripts.
> This will improve the out-of-box experience of users that otherwise have to 
> manually set {{HADOOP_CLASSPATH}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5521: [FLINK-8599] Improve the failure behavior of the F...

2018-02-18 Thread ChengzhiZhao

GitHub user ChengzhiZhao opened a pull request:

https://github.com/apache/flink/pull/5521

[FLINK-8599] Improve the failure behavior of the FileInputFormat for â¦

## What is the purpose of the change

This pull request is intent to improve the failure behavior of the 
ContinuousFileReader, currently if a bad file (for example, a different schema 
been dropped in this folder) came to the path and flink will do several retries.
However, since the file path persist in the checkpoint, when people tried 
to resume from external checkpoint, it threw the following error on no file 
been found and the process cannot move forward.

`java.io.IOException: Error opening the Input Split s3a://myfile [0,904]: 
No such file or directory: s3a://myfile`

The change is to check if the path exist before open the file, if error 
occurs and bad file removed, flink should resume the process and continue.

## Brief change log
- *Add a file exist check before open the file *

## Verifying this change
- *Manually verified the change by introduce a bad file while continuously 
monitoring the folder, after remove the bad file, the process continued.*

## Does this pull request potentially affect one of the following parts:
  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ChengzhiZhao/flink 
Improve_failure_behavior_FileInputFormat

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5521.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5521


commit 6fa8ef212c536acee56b2e9831ec92d1059449ff
Author: Chengzhi Zhao 
Date:   2018-02-18T18:23:32Z

[FLINK-8599] Improve the failure behavior of the FileInputFormat for bad 
files




---

[jira] [Closed] (FLINK-8692) Mistake in MyMapFunction code snippet

2018-02-18 Thread Chesnay Schepler (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-8692.
---
Resolution: Fixed

master: 0644f4f77dde4532714fb60bf8c646365fca5fc0
1.4: 528317c8f23a7f20493510e57c21c72a1fc42b3d

> Mistake in MyMapFunction code snippet
> -
>
> Key: FLINK-8692
> URL: https://issues.apache.org/jira/browse/FLINK-8692
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.5.0, 1.4.1
>Reporter: Matt Hagen
>Assignee: Chesnay Schepler
>Priority: Trivial
> Fix For: 1.5.0, 1.4.2
>
> Attachments: my-map-function.png
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The MyMapFunction code snippet on the [Basic API 
> Concepts|https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/api_concepts.html]
>  page has an extra parenthesis. Just remove the last parenthesis. See the 
> attached screenshot. Thanks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8692) Mistake in MyMapFunction code snippet

2018-02-18 Thread Chesnay Schepler (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-8692:

Labels:   (was: documentation)

> Mistake in MyMapFunction code snippet
> -
>
> Key: FLINK-8692
> URL: https://issues.apache.org/jira/browse/FLINK-8692
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.5.0, 1.4.1
>Reporter: Matt Hagen
>Assignee: Chesnay Schepler
>Priority: Trivial
> Fix For: 1.5.0, 1.4.2
>
> Attachments: my-map-function.png
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The MyMapFunction code snippet on the [Basic API 
> Concepts|https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/api_concepts.html]
>  page has an extra parenthesis. Just remove the last parenthesis. See the 
> attached screenshot. Thanks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8692) Mistake in MyMapFunction code snippet

2018-02-18 Thread Chesnay Schepler (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-8692:

Affects Version/s: 1.5.0

> Mistake in MyMapFunction code snippet
> -
>
> Key: FLINK-8692
> URL: https://issues.apache.org/jira/browse/FLINK-8692
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.5.0, 1.4.1
>Reporter: Matt Hagen
>Assignee: Chesnay Schepler
>Priority: Trivial
> Fix For: 1.5.0, 1.4.2
>
> Attachments: my-map-function.png
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The MyMapFunction code snippet on the [Basic API 
> Concepts|https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/api_concepts.html]
>  page has an extra parenthesis. Just remove the last parenthesis. See the 
> attached screenshot. Thanks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-8692) Mistake in MyMapFunction code snippet

2018-02-18 Thread Chesnay Schepler (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler reassigned FLINK-8692:
---

Assignee: Chesnay Schepler

> Mistake in MyMapFunction code snippet
> -
>
> Key: FLINK-8692
> URL: https://issues.apache.org/jira/browse/FLINK-8692
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.5.0, 1.4.1
>Reporter: Matt Hagen
>Assignee: Chesnay Schepler
>Priority: Trivial
> Fix For: 1.5.0, 1.4.2
>
> Attachments: my-map-function.png
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The MyMapFunction code snippet on the [Basic API 
> Concepts|https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/api_concepts.html]
>  page has an extra parenthesis. Just remove the last parenthesis. See the 
> attached screenshot. Thanks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8692) Mistake in MyMapFunction code snippet

2018-02-18 Thread Chesnay Schepler (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-8692:

Fix Version/s: 1.4.2
   1.5.0

> Mistake in MyMapFunction code snippet
> -
>
> Key: FLINK-8692
> URL: https://issues.apache.org/jira/browse/FLINK-8692
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.5.0, 1.4.1
>Reporter: Matt Hagen
>Assignee: Chesnay Schepler
>Priority: Trivial
> Fix For: 1.5.0, 1.4.2
>
> Attachments: my-map-function.png
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The MyMapFunction code snippet on the [Basic API 
> Concepts|https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/api_concepts.html]
>  page has an extra parenthesis. Just remove the last parenthesis. See the 
> attached screenshot. Thanks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8693) initPythonInterpreter is called twice with same arguments in InterpreterUtils#initAndExecPythonScript

2018-02-18 Thread Chesnay Schepler (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-8693.
---
Resolution: Fixed

master: 8cea2f6576da713e5625e00ef2dbe34a410a1df1

> initPythonInterpreter is called twice with same arguments in 
> InterpreterUtils#initAndExecPythonScript
> -
>
> Key: FLINK-8693
> URL: https://issues.apache.org/jira/browse/FLINK-8693
> Project: Flink
>  Issue Type: Bug
>  Components: Python API
>Affects Versions: 1.5.0
>Reporter: Ted Yu
>Assignee: Chesnay Schepler
>Priority: Trivial
> Fix For: 1.5.0
>
>
> {code}
> initPythonInterpreter(fullArgs, scriptDirectory.toUri().getPath(), 
> scriptName);
> PythonInterpreter pythonInterpreter = initPythonInterpreter(fullArgs, 
> scriptDirectory.toUri().getPath(), scriptName);
> {code}
> The two calls to initPythonInterpreter() have the same parameters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8693) initPythonInterpreter is called twice with same arguments in InterpreterUtils#initAndExecPythonScript

2018-02-18 Thread Chesnay Schepler (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-8693:

Priority: Trivial  (was: Minor)

> initPythonInterpreter is called twice with same arguments in 
> InterpreterUtils#initAndExecPythonScript
> -
>
> Key: FLINK-8693
> URL: https://issues.apache.org/jira/browse/FLINK-8693
> Project: Flink
>  Issue Type: Bug
>  Components: Python API
>Affects Versions: 1.5.0
>Reporter: Ted Yu
>Assignee: Chesnay Schepler
>Priority: Trivial
> Fix For: 1.5.0
>
>
> {code}
> initPythonInterpreter(fullArgs, scriptDirectory.toUri().getPath(), 
> scriptName);
> PythonInterpreter pythonInterpreter = initPythonInterpreter(fullArgs, 
> scriptDirectory.toUri().getPath(), scriptName);
> {code}
> The two calls to initPythonInterpreter() have the same parameters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8693) initPythonInterpreter is called twice with same arguments in InterpreterUtils#initAndExecPythonScript

2018-02-18 Thread Chesnay Schepler (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-8693:

Fix Version/s: 1.5.0

> initPythonInterpreter is called twice with same arguments in 
> InterpreterUtils#initAndExecPythonScript
> -
>
> Key: FLINK-8693
> URL: https://issues.apache.org/jira/browse/FLINK-8693
> Project: Flink
>  Issue Type: Bug
>  Components: Python API
>Affects Versions: 1.5.0
>Reporter: Ted Yu
>Assignee: Chesnay Schepler
>Priority: Minor
> Fix For: 1.5.0
>
>
> {code}
> initPythonInterpreter(fullArgs, scriptDirectory.toUri().getPath(), 
> scriptName);
> PythonInterpreter pythonInterpreter = initPythonInterpreter(fullArgs, 
> scriptDirectory.toUri().getPath(), scriptName);
> {code}
> The two calls to initPythonInterpreter() have the same parameters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8693) initPythonInterpreter is called twice with same arguments in InterpreterUtils#initAndExecPythonScript

2018-02-18 Thread Chesnay Schepler (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-8693:

Component/s: Python API

> initPythonInterpreter is called twice with same arguments in 
> InterpreterUtils#initAndExecPythonScript
> -
>
> Key: FLINK-8693
> URL: https://issues.apache.org/jira/browse/FLINK-8693
> Project: Flink
>  Issue Type: Bug
>  Components: Python API
>Affects Versions: 1.5.0
>Reporter: Ted Yu
>Assignee: Chesnay Schepler
>Priority: Minor
> Fix For: 1.5.0
>
>
> {code}
> initPythonInterpreter(fullArgs, scriptDirectory.toUri().getPath(), 
> scriptName);
> PythonInterpreter pythonInterpreter = initPythonInterpreter(fullArgs, 
> scriptDirectory.toUri().getPath(), scriptName);
> {code}
> The two calls to initPythonInterpreter() have the same parameters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8693) initPythonInterpreter is called twice with same arguments in InterpreterUtils#initAndExecPythonScript

2018-02-18 Thread Chesnay Schepler (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-8693:

Affects Version/s: 1.5.0

> initPythonInterpreter is called twice with same arguments in 
> InterpreterUtils#initAndExecPythonScript
> -
>
> Key: FLINK-8693
> URL: https://issues.apache.org/jira/browse/FLINK-8693
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Ted Yu
>Assignee: Chesnay Schepler
>Priority: Minor
>
> {code}
> initPythonInterpreter(fullArgs, scriptDirectory.toUri().getPath(), 
> scriptName);
> PythonInterpreter pythonInterpreter = initPythonInterpreter(fullArgs, 
> scriptDirectory.toUri().getPath(), scriptName);
> {code}
> The two calls to initPythonInterpreter() have the same parameters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-8693) initPythonInterpreter is called twice with same arguments in InterpreterUtils#initAndExecPythonScript

2018-02-18 Thread Chesnay Schepler (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler reassigned FLINK-8693:
---

Assignee: Chesnay Schepler

> initPythonInterpreter is called twice with same arguments in 
> InterpreterUtils#initAndExecPythonScript
> -
>
> Key: FLINK-8693
> URL: https://issues.apache.org/jira/browse/FLINK-8693
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chesnay Schepler
>Priority: Minor
>
> {code}
> initPythonInterpreter(fullArgs, scriptDirectory.toUri().getPath(), 
> scriptName);
> PythonInterpreter pythonInterpreter = initPythonInterpreter(fullArgs, 
> scriptDirectory.toUri().getPath(), scriptName);
> {code}
> The two calls to initPythonInterpreter() have the same parameters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8693) initPythonInterpreter is called twice with same arguments in InterpreterUtils#initAndExecPythonScript

2018-02-18 Thread Ted Yu (JIRA)

Ted Yu created FLINK-8693:
-

 Summary: initPythonInterpreter is called twice with same arguments 
in InterpreterUtils#initAndExecPythonScript
 Key: FLINK-8693
 URL: https://issues.apache.org/jira/browse/FLINK-8693
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


{code}
initPythonInterpreter(fullArgs, scriptDirectory.toUri().getPath(), 
scriptName);

PythonInterpreter pythonInterpreter = initPythonInterpreter(fullArgs, 
scriptDirectory.toUri().getPath(), scriptName);
{code}
The two calls to initPythonInterpreter() have the same parameters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8692) Mistake in MyMapFunction code snippet

2018-02-18 Thread Matt Hagen (JIRA)

Matt Hagen created FLINK-8692:
-

 Summary: Mistake in MyMapFunction code snippet
 Key: FLINK-8692
 URL: https://issues.apache.org/jira/browse/FLINK-8692
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.4.1
Reporter: Matt Hagen
 Attachments: my-map-function.png

The MyMapFunction code snippet on the [Basic API 
Concepts|https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/api_concepts.html]
 page has an extra parenthesis. Just remove the last parenthesis. See the 
attached screenshot. Thanks.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5520: [FLINK-8599] Improve the failure behavior of the F...

2018-02-18 Thread ChengzhiZhao

Github user ChengzhiZhao closed the pull request at:

https://github.com/apache/flink/pull/5520


---

[jira] [Commented] (FLINK-8599) Improve the failure behavior of the FileInputFormat for bad files

2018-02-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368614#comment-16368614
 ] 

ASF GitHub Bot commented on FLINK-8599:
---

Github user ChengzhiZhao closed the pull request at:

https://github.com/apache/flink/pull/5520


> Improve the failure behavior of the FileInputFormat for bad files
> -
>
> Key: FLINK-8599
> URL: https://issues.apache.org/jira/browse/FLINK-8599
> Project: Flink
>  Issue Type: New Feature
>  Components: DataStream API
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Chengzhi Zhao
>Priority: Major
>
> So we have a s3 path that flink is monitoring that path to see new files 
> available.
> {code:java}
> val avroInputStream_activity = env.readFile(format, path, 
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1)  {code}
>  
> I am doing both internal and external check pointing and let's say there is a 
> bad file (for example, a different schema been dropped in this folder) came 
> to the path and flink will do several retries. I want to take those bad files 
> and let the process continue. However, since the file path persist in the 
> checkpoint, when I try to resume from external checkpoint, it threw the 
> following error on no file been found.
>  
> {code:java}
> java.io.IOException: Error opening the Input Split s3a://myfile [0,904]: No 
> such file or directory: s3a://myfile{code}
>  
> As [~fhue...@gmail.com] suggested, we could check if a path exists and before 
> trying to read a file and ignore the input split instead of throwing an 
> exception and causing a failure.
>  
> Also, I am thinking about add an error output for bad files as an option to 
> users. So if there is any bad files exist we could move them in a separated 
> path and do further analysis. 
>  
> Not sure how people feel about it, but I'd like to contribute on it if people 
> think this can be an improvement. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-8684) Rework MesosTaskManagerParameters#MESOS_RM_TASKS_SLOTS

2018-02-18 Thread Chesnay Schepler (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler reassigned FLINK-8684:
---

Assignee: Chesnay Schepler

> Rework MesosTaskManagerParameters#MESOS_RM_TASKS_SLOTS
> --
>
> Key: FLINK-8684
> URL: https://issues.apache.org/jira/browse/FLINK-8684
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation, Mesos
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Fix For: 1.5.0
>
>
> Currently, {{MesosTaskManagerParameters#MESOS_RM_TASKS_SLOTS}} mimics 
> {{TaskManagerOptions#NUM_TASK_SLOTS}}:
> {code:java}
> public static final ConfigOption MESOS_RM_TASKS_SLOTS =
>   key(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS)
>   .defaultValue(1);
> public static final ConfigOption NUM_TASK_SLOTS =
>   key("taskmanager.numberOfTaskSlots")
>   .defaultValue(1)
>   .withDescription("...");
> {code}
> This pattern is problematic as this creates 2 documentation entries for 
> {{taskmanager.numberOfTaskSlots}} with different descriptions, and opens the 
> potential for different defaults. Ultimately this causes the documentation to 
> become ambiguous.
> I thus propose to either outright remove this option or turn it into an 
> actual alias:
> {code:java}
> public static final ConfigOption MESOS_RM_TASKS_SLOTS =
> TaskManagerOptions.NUM_TASK_SLOTS;
> {code}
> As a side-effect of FLINK-8683 we can ensure that no differing config options 
> exist for a given key.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-8689) Add runtime support of distinct filter using MapView for GenerateAggregation

2018-02-18 Thread Rong Rong (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rong Rong reassigned FLINK-8689:


Assignee: Rong Rong

> Add runtime support of distinct filter using MapView for GenerateAggregation
> 
>
> Key: FLINK-8689
> URL: https://issues.apache.org/jira/browse/FLINK-8689
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Rong Rong
>Assignee: Rong Rong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-8690) Update logical rule set to generate FlinkLogicalAggregate explicitly allow distinct operator

2018-02-18 Thread Rong Rong (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rong Rong reassigned FLINK-8690:


Assignee: Rong Rong

> Update logical rule set to generate FlinkLogicalAggregate explicitly allow 
> distinct operator 
> -
>
> Key: FLINK-8690
> URL: https://issues.apache.org/jira/browse/FLINK-8690
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Rong Rong
>Assignee: Rong Rong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8691) Update table API to support distinct operator on data stream

2018-02-18 Thread Rong Rong (JIRA)

Rong Rong created FLINK-8691:


 Summary: Update table API to support distinct operator on data 
stream
 Key: FLINK-8691
 URL: https://issues.apache.org/jira/browse/FLINK-8691
 Project: Flink
  Issue Type: Sub-task
Reporter: Rong Rong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8690) Update logical rule set to generate FlinkLogicalAggregate explicitly allow distinct operator

2018-02-18 Thread Rong Rong (JIRA)

Rong Rong created FLINK-8690:


 Summary: Update logical rule set to generate FlinkLogicalAggregate 
explicitly allow distinct operator 
 Key: FLINK-8690
 URL: https://issues.apache.org/jira/browse/FLINK-8690
 Project: Flink
  Issue Type: Sub-task
Reporter: Rong Rong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8689) Add runtime support of distinct filter using MapView for GenerateAggregation

2018-02-18 Thread Rong Rong (JIRA)

Rong Rong created FLINK-8689:


 Summary: Add runtime support of distinct filter using MapView for 
GenerateAggregation
 Key: FLINK-8689
 URL: https://issues.apache.org/jira/browse/FLINK-8689
 Project: Flink
  Issue Type: Sub-task
Reporter: Rong Rong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8688) Enable distinct aggregation for data stream on Table/SQL API

2018-02-18 Thread Rong Rong (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368585#comment-16368585
 ] 

Rong Rong commented on FLINK-8688:
--

Related distinct aggregate feature request JIRA

> Enable distinct aggregation for data stream on Table/SQL API
> 
>
> Key: FLINK-8688
> URL: https://issues.apache.org/jira/browse/FLINK-8688
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API  SQL
>Reporter: Rong Rong
>Assignee: Rong Rong
>Priority: Major
>
> Distinct aggregation is not currently supported on data stream with Table/SQL 
> API. This is an umbrella task for enabling distinct aggregation in various 
> use cases.
> Discussion doc can be found here: 
> https://docs.google.com/document/d/1zj6OA-K2hi7ah8Fo-xTQB-mVmYfm6LsN2_NHgTCVmJI/edit?usp=sharing
>  
> Distinct aggregation is a very important feature in SQL processing and there 
> are many JIRAs currently open with various use cases. The goal is to create 
> one solution to both unbounded and bounded distinct aggregation on data 
> stream so that it can easily be extended to support these use cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8688) Enable distinct aggregation for data stream on Table/SQL API

2018-02-18 Thread Rong Rong (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rong Rong updated FLINK-8688:
-
Description: 
Distinct aggregation is not currently supported on data stream with Table/SQL 
API. This is an umbrella task for enabling distinct aggregation in various use 
cases.

Discussion doc can be found here: 
https://docs.google.com/document/d/1zj6OA-K2hi7ah8Fo-xTQB-mVmYfm6LsN2_NHgTCVmJI/edit?usp=sharing
 

Distinct aggregation is a very important feature in SQL processing and there 
are many JIRAs currently open with various use cases. The goal is to create one 
solution to both unbounded and bounded distinct aggregation on data stream so 
that it can easily be extended to support these use cases.

  was:
Distinct aggregation is not currently supported on data stream with Table/SQL 
API. This is an umbrella task for enabling distinct aggregation in various use 
cases.

Discussion doc can be found [ here | 
[https://docs.google.com/document/d/1zj6OA-K2hi7ah8Fo-xTQB-mVmYfm6LsN2_NHgTCVmJI/edit?usp=sharing]
 ]

 

Distinct aggregation is a very important feature in SQL processing and there 
are many JIRAs currently open with various use cases. The goal is to create one 
solution to both unbounded and bounded distinct aggregation on data stream so 
that it can easily be extended to support these use cases.


> Enable distinct aggregation for data stream on Table/SQL API
> 
>
> Key: FLINK-8688
> URL: https://issues.apache.org/jira/browse/FLINK-8688
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API  SQL
>Reporter: Rong Rong
>Assignee: Rong Rong
>Priority: Major
>
> Distinct aggregation is not currently supported on data stream with Table/SQL 
> API. This is an umbrella task for enabling distinct aggregation in various 
> use cases.
> Discussion doc can be found here: 
> https://docs.google.com/document/d/1zj6OA-K2hi7ah8Fo-xTQB-mVmYfm6LsN2_NHgTCVmJI/edit?usp=sharing
>  
> Distinct aggregation is a very important feature in SQL processing and there 
> are many JIRAs currently open with various use cases. The goal is to create 
> one solution to both unbounded and bounded distinct aggregation on data 
> stream so that it can easily be extended to support these use cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-8688) Enable distinct aggregation for data stream on Table/SQL API

2018-02-18 Thread Rong Rong (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rong Rong updated FLINK-8688:
-
Description: 
Distinct aggregation is not currently supported on data stream with Table/SQL 
API. This is an umbrella task for enabling distinct aggregation in various use 
cases.

Discussion doc can be found [ here | 
[https://docs.google.com/document/d/1zj6OA-K2hi7ah8Fo-xTQB-mVmYfm6LsN2_NHgTCVmJI/edit?usp=sharing]
 ]

 

Distinct aggregation is a very important feature in SQL processing and there 
are many JIRAs currently open with various use cases. The goal is to create one 
solution to both unbounded and bounded distinct aggregation on data stream so 
that it can easily be extended to support these use cases.

  was:
Distinct aggregation is not currently supported on data stream with Table/SQL 
API. This is an umbrella task for enabling distinct aggregation in various use 
cases.

Discussion doc can be found 
[here|[https://docs.google.com/document/d/1zj6OA-K2hi7ah8Fo-xTQB-mVmYfm6LsN2_NHgTCVmJI/edit?usp=sharing]]

 

Distinct aggregation is a very important feature in SQL processing and there 
are many JIRAs currently open with various use cases. The goal is to create one 
solution to both unbounded and bounded distinct aggregation on data stream so 
that it can easily be extended to support these use cases.


> Enable distinct aggregation for data stream on Table/SQL API
> 
>
> Key: FLINK-8688
> URL: https://issues.apache.org/jira/browse/FLINK-8688
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API  SQL
>Reporter: Rong Rong
>Assignee: Rong Rong
>Priority: Major
>
> Distinct aggregation is not currently supported on data stream with Table/SQL 
> API. This is an umbrella task for enabling distinct aggregation in various 
> use cases.
> Discussion doc can be found [ here | 
> [https://docs.google.com/document/d/1zj6OA-K2hi7ah8Fo-xTQB-mVmYfm6LsN2_NHgTCVmJI/edit?usp=sharing]
>  ]
>  
> Distinct aggregation is a very important feature in SQL processing and there 
> are many JIRAs currently open with various use cases. The goal is to create 
> one solution to both unbounded and bounded distinct aggregation on data 
> stream so that it can easily be extended to support these use cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8688) Enable distinct aggregation for data stream on Table/SQL API

2018-02-18 Thread Rong Rong (JIRA)

Rong Rong created FLINK-8688:


 Summary: Enable distinct aggregation for data stream on Table/SQL 
API
 Key: FLINK-8688
 URL: https://issues.apache.org/jira/browse/FLINK-8688
 Project: Flink
  Issue Type: Improvement
  Components: Table API  SQL
Reporter: Rong Rong
Assignee: Rong Rong


Distinct aggregation is not currently supported on data stream with Table/SQL 
API. This is an umbrella task for enabling distinct aggregation in various use 
cases.

Discussion doc can be found 
[here|[https://docs.google.com/document/d/1zj6OA-K2hi7ah8Fo-xTQB-mVmYfm6LsN2_NHgTCVmJI/edit?usp=sharing]]

 

Distinct aggregation is a very important feature in SQL processing and there 
are many JIRAs currently open with various use cases. The goal is to create one 
solution to both unbounded and bounded distinct aggregation on data stream so 
that it can easily be extended to support these use cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8684) Rework MesosTaskManagerParameters#MESOS_RM_TASKS_SLOTS

2018-02-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368557#comment-16368557
 ] 

ASF GitHub Bot commented on FLINK-8684:
---

Github user zhangminglei closed the pull request at:

https://github.com/apache/flink/pull/5519


> Rework MesosTaskManagerParameters#MESOS_RM_TASKS_SLOTS
> --
>
> Key: FLINK-8684
> URL: https://issues.apache.org/jira/browse/FLINK-8684
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation, Mesos
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Priority: Major
> Fix For: 1.5.0
>
>
> Currently, {{MesosTaskManagerParameters#MESOS_RM_TASKS_SLOTS}} mimics 
> {{TaskManagerOptions#NUM_TASK_SLOTS}}:
> {code:java}
> public static final ConfigOption MESOS_RM_TASKS_SLOTS =
>   key(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS)
>   .defaultValue(1);
> public static final ConfigOption NUM_TASK_SLOTS =
>   key("taskmanager.numberOfTaskSlots")
>   .defaultValue(1)
>   .withDescription("...");
> {code}
> This pattern is problematic as this creates 2 documentation entries for 
> {{taskmanager.numberOfTaskSlots}} with different descriptions, and opens the 
> potential for different defaults. Ultimately this causes the documentation to 
> become ambiguous.
> I thus propose to either outright remove this option or turn it into an 
> actual alias:
> {code:java}
> public static final ConfigOption MESOS_RM_TASKS_SLOTS =
> TaskManagerOptions.NUM_TASK_SLOTS;
> {code}
> As a side-effect of FLINK-8683 we can ensure that no differing config options 
> exist for a given key.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5519: [FLINK-8684] [config] Rework MesosTaskManagerParam...

2018-02-18 Thread zhangminglei

Github user zhangminglei closed the pull request at:

https://github.com/apache/flink/pull/5519


---

[jira] [Commented] (FLINK-8599) Improve the failure behavior of the FileInputFormat for bad files

2018-02-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368547#comment-16368547
 ] 

ASF GitHub Bot commented on FLINK-8599:
---

GitHub user ChengzhiZhao opened a pull request:

https://github.com/apache/flink/pull/5520

[FLINK-8599] Improve the failure behavior of the FileInputFormat for bad 
files

## What is the purpose of the change

This pull request is intent to improve the failure behavior of the 
FileInputFormat, currently if a bad file (for example, a different schema been 
dropped in this folder) came to the path and flink will do several retries.
However, since the file path persist in the checkpoint, when people tried 
to resume from external checkpoint, it threw the following error on no file 
been found and the process cannot move forward.

`java.io.IOException: Error opening the Input Split s3a://myfile [0,904]: 
No such file or directory: s3a://myfile`

The change is to check if the path exist before open the file, if error 
occurs and bad file removed, flink should resume the process and continue.

## Brief change log
- *Add a file exist check before open the file *

## Verifying this change
- *Manually verified the change by introduce a bad file while continuously 
monitoring the folder, after remove the bad file, the process continued.*

## Does this pull request potentially affect one of the following parts:
  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ChengzhiZhao/flink 
Improve_failure_behavior_ContinuousFileReaderOperator

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5520.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5520


commit e1605306d5f4a7c7c52eb1e1f3d213ea9872c71b
Author: Chengzhi Zhao 
Date:   2018-02-16T16:03:01Z

[FLINK-8599] Improve the failure behavior of the 
ContinuousFileReaderOperator

commit d8074e5141e0e6dc9f66e1fab8275cec3803904f
Author: Chengzhi Zhao 
Date:   2018-02-17T21:04:59Z

[FLINK-8599] break while loop

commit 27c031466e1ea2d31cace5cd6a7f33f0d4c896c6
Author: Chengzhi Zhao 
Date:   2018-02-18T03:14:05Z

[FLINK-8599] Check file exists in FileInputFormat

commit a087587d12c5118f7ba8e55692dae6f3e95ef50b
Author: Chengzhi Zhao 
Date:   2018-02-18T04:13:50Z

[FLINK-8599] Add debug




> Improve the failure behavior of the FileInputFormat for bad files
> -
>
> Key: FLINK-8599
> URL: https://issues.apache.org/jira/browse/FLINK-8599
> Project: Flink
>  Issue Type: New Feature
>  Components: DataStream API
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Chengzhi Zhao
>Priority: Major
>
> So we have a s3 path that flink is monitoring that path to see new files 
> available.
> {code:java}
> val avroInputStream_activity = env.readFile(format, path, 
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1)  {code}
>  
> I am doing both internal and external check pointing and let's say there is a 
> bad file (for example, a different schema been dropped in this folder) came 
> to the path and flink will do several retries. I want to take those bad files 
> and let the process continue. However, since the file path persist in the 
> checkpoint, when I try to resume from external checkpoint, it threw the 
> following error on no file been found.
>  
> {code:java}
> java.io.IOException: Error opening the Input Split s3a://myfile [0,904]: No 
> such file or directory: s3a://myfile{code}
>  
> As [~fhue...@gmail.com] suggested, we could check if a path exists and before 
> trying to read a file and ignore the input split instead of throwing an 
> exception and causing a failure.
>  
> Also, I am thinking about add an error output for bad files as an option to 
> users. So if there is any bad files exist we could move them in a separated 
> path and do further analysis. 
>  
> Not sure how people feel about it, but I'd like to contribute on it if people 
>

[GitHub] flink pull request #5520: [FLINK-8599] Improve the failure behavior of the F...

2018-02-18 Thread ChengzhiZhao

GitHub user ChengzhiZhao opened a pull request:

https://github.com/apache/flink/pull/5520

[FLINK-8599] Improve the failure behavior of the FileInputFormat for bad 
files

## What is the purpose of the change

This pull request is intent to improve the failure behavior of the 
FileInputFormat, currently if a bad file (for example, a different schema been 
dropped in this folder) came to the path and flink will do several retries.
However, since the file path persist in the checkpoint, when people tried 
to resume from external checkpoint, it threw the following error on no file 
been found and the process cannot move forward.

`java.io.IOException: Error opening the Input Split s3a://myfile [0,904]: 
No such file or directory: s3a://myfile`

The change is to check if the path exist before open the file, if error 
occurs and bad file removed, flink should resume the process and continue.

## Brief change log
- *Add a file exist check before open the file *

## Verifying this change
- *Manually verified the change by introduce a bad file while continuously 
monitoring the folder, after remove the bad file, the process continued.*

## Does this pull request potentially affect one of the following parts:
  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ChengzhiZhao/flink 
Improve_failure_behavior_ContinuousFileReaderOperator

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5520.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5520


commit e1605306d5f4a7c7c52eb1e1f3d213ea9872c71b
Author: Chengzhi Zhao 
Date:   2018-02-16T16:03:01Z

[FLINK-8599] Improve the failure behavior of the 
ContinuousFileReaderOperator

commit d8074e5141e0e6dc9f66e1fab8275cec3803904f
Author: Chengzhi Zhao 
Date:   2018-02-17T21:04:59Z

[FLINK-8599] break while loop

commit 27c031466e1ea2d31cace5cd6a7f33f0d4c896c6
Author: Chengzhi Zhao 
Date:   2018-02-18T03:14:05Z

[FLINK-8599] Check file exists in FileInputFormat

commit a087587d12c5118f7ba8e55692dae6f3e95ef50b
Author: Chengzhi Zhao 
Date:   2018-02-18T04:13:50Z

[FLINK-8599] Add debug




---

[jira] [Updated] (FLINK-8599) Improve the failure behavior of the FileInputFormat for bad files

2018-02-18 Thread Chengzhi Zhao (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chengzhi Zhao updated FLINK-8599:
-
Summary: Improve the failure behavior of the FileInputFormat for bad files  
(was: Improve the failure behavior of the ContinuousFileReaderOperator for bad 
files)

> Improve the failure behavior of the FileInputFormat for bad files
> -
>
> Key: FLINK-8599
> URL: https://issues.apache.org/jira/browse/FLINK-8599
> Project: Flink
>  Issue Type: New Feature
>  Components: DataStream API
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Chengzhi Zhao
>Priority: Major
>
> So we have a s3 path that flink is monitoring that path to see new files 
> available.
> {code:java}
> val avroInputStream_activity = env.readFile(format, path, 
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1)  {code}
>  
> I am doing both internal and external check pointing and let's say there is a 
> bad file (for example, a different schema been dropped in this folder) came 
> to the path and flink will do several retries. I want to take those bad files 
> and let the process continue. However, since the file path persist in the 
> checkpoint, when I try to resume from external checkpoint, it threw the 
> following error on no file been found.
>  
> {code:java}
> java.io.IOException: Error opening the Input Split s3a://myfile [0,904]: No 
> such file or directory: s3a://myfile{code}
>  
> As [~fhue...@gmail.com] suggested, we could check if a path exists and before 
> trying to read a file and ignore the input split instead of throwing an 
> exception and causing a failure.
>  
> Also, I am thinking about add an error output for bad files as an option to 
> users. So if there is any bad files exist we could move them in a separated 
> path and do further analysis. 
>  
> Not sure how people feel about it, but I'd like to contribute on it if people 
> think this can be an improvement. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8546) Respect savepoint settings and recover from latest checkpoint in Flip-6

2018-02-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368494#comment-16368494
 ] 

ASF GitHub Bot commented on FLINK-8546:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5444


> Respect savepoint settings and recover from latest checkpoint in Flip-6
> ---
>
> Key: FLINK-8546
> URL: https://issues.apache.org/jira/browse/FLINK-8546
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> The {{JobMaster}} should respect savepoints and recover from the latest 
> checkpoint if possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8614) Enable Flip-6 per default

2018-02-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368493#comment-16368493
 ] 

ASF GitHub Bot commented on FLINK-8614:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5437


> Enable Flip-6 per default
> -
>
> Key: FLINK-8614
> URL: https://issues.apache.org/jira/browse/FLINK-8614
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> After adding the FLINK-8471, the next step is to enable Flip-6 per default by 
> setting the configuration switch to {{flip6}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8673) Don't let JobManagerRunner shut down itself

2018-02-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368492#comment-16368492
 ] 

ASF GitHub Bot commented on FLINK-8673:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5510


> Don't let JobManagerRunner shut down itself
> ---
>
> Key: FLINK-8673
> URL: https://issues.apache.org/jira/browse/FLINK-8673
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Currently, the {{JobManagerRunner}} is allowed to shut down itself in case of 
> a job completion. This, however, can cause problems when the {{Dispatcher}} 
> receives a request for a {{JobMaster}}. If the {{Dispatcher}} is not told 
> about the shut down of the {{JobMaster}} then it might still try to send 
> requests to it. This will lead to time outs.
> It would be better to simply let the {{JobManagerRunner}} not shut down 
> itself and defer it to the owner (the {{Dispatcher}}). We can do this by 
> listening on the {{JobManagerRunner#resultFuture}} which is completed by the 
> {{JobManagerRunner}} in case of a successful job completion or a failure. 
> That way we could also get rid of the {{OnCompletionActions}} and the 
> {{FatalErrorHandler}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8653) Remove slot request timeout from SlotPool

2018-02-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368495#comment-16368495
 ] 

ASF GitHub Bot commented on FLINK-8653:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5483


> Remove slot request timeout from SlotPool
> -
>
> Key: FLINK-8653
> URL: https://issues.apache.org/jira/browse/FLINK-8653
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> After addressing FLINK-8643, we can further simplify the {{SlotPool}} by 
> replacing the internal slot request timeout by the timeout given to 
> {{SlotPool#allocateSlot}}. Since this request will timeout on the 
> {{ProviderAndOwner}} side anyway, we should do the same on the {{SlotPool}} 
> side.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5437: [FLINK-8614] [flip6] Activate Flip-6 mode per defa...

2018-02-18 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5437


---

[GitHub] flink pull request #5483: [FLINK-8653] [flip6] Remove internal slot request ...

2018-02-18 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5483


---

[jira] [Closed] (FLINK-8673) Don't let JobManagerRunner shut down itself

2018-02-18 Thread Till Rohrmann (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-8673.

Resolution: Fixed

Fixed via 075f5b6930b86e110a32e290229b9800be72a3a7

> Don't let JobManagerRunner shut down itself
> ---
>
> Key: FLINK-8673
> URL: https://issues.apache.org/jira/browse/FLINK-8673
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Currently, the {{JobManagerRunner}} is allowed to shut down itself in case of 
> a job completion. This, however, can cause problems when the {{Dispatcher}} 
> receives a request for a {{JobMaster}}. If the {{Dispatcher}} is not told 
> about the shut down of the {{JobMaster}} then it might still try to send 
> requests to it. This will lead to time outs.
> It would be better to simply let the {{JobManagerRunner}} not shut down 
> itself and defer it to the owner (the {{Dispatcher}}). We can do this by 
> listening on the {{JobManagerRunner#resultFuture}} which is completed by the 
> {{JobManagerRunner}} in case of a successful job completion or a failure. 
> That way we could also get rid of the {{OnCompletionActions}} and the 
> {{FatalErrorHandler}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8614) Enable Flip-6 per default

2018-02-18 Thread Till Rohrmann (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-8614.

Resolution: Fixed

Fixed via ab8316f31a4de0648ab3ffa9f19fcd419aaa3bb9

> Enable Flip-6 per default
> -
>
> Key: FLINK-8614
> URL: https://issues.apache.org/jira/browse/FLINK-8614
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> After adding the FLINK-8471, the next step is to enable Flip-6 per default by 
> setting the configuration switch to {{flip6}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5510: [FLINK-8673] [flip6] Use JobManagerRunner#resultFu...

2018-02-18 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5510


---

[GitHub] flink pull request #5444: [FLINK-8546] [flip6] Respect savepoints and restor...

2018-02-18 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5444


---

[jira] [Closed] (FLINK-8546) Respect savepoint settings and recover from latest checkpoint in Flip-6

2018-02-18 Thread Till Rohrmann (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-8546.

Resolution: Fixed

Fixed via 3a4e89643d7d7642dde9b5644491f261d4d545bd

> Respect savepoint settings and recover from latest checkpoint in Flip-6
> ---
>
> Key: FLINK-8546
> URL: https://issues.apache.org/jira/browse/FLINK-8546
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> The {{JobMaster}} should respect savepoints and recover from the latest 
> checkpoint if possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8653) Remove slot request timeout from SlotPool

2018-02-18 Thread Till Rohrmann (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-8653.

Resolution: Fixed

Fixed via d8a8866973f7e0463047963e6b242cdc2cb82fec

> Remove slot request timeout from SlotPool
> -
>
> Key: FLINK-8653
> URL: https://issues.apache.org/jira/browse/FLINK-8653
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> After addressing FLINK-8643, we can further simplify the {{SlotPool}} by 
> replacing the internal slot request timeout by the timeout given to 
> {{SlotPool#allocateSlot}}. Since this request will timeout on the 
> {{ProviderAndOwner}} side anyway, we should do the same on the {{SlotPool}} 
> side.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8684) Rework MesosTaskManagerParameters#MESOS_RM_TASKS_SLOTS

2018-02-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368482#comment-16368482
 ] 

ASF GitHub Bot commented on FLINK-8684:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5519
  
Correct, I've included a fix for FLINK-8684 in the 
[PR](https://github.com/apache/flink/pull/5515) for FLINK-8683.


> Rework MesosTaskManagerParameters#MESOS_RM_TASKS_SLOTS
> --
>
> Key: FLINK-8684
> URL: https://issues.apache.org/jira/browse/FLINK-8684
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation, Mesos
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Priority: Major
> Fix For: 1.5.0
>
>
> Currently, {{MesosTaskManagerParameters#MESOS_RM_TASKS_SLOTS}} mimics 
> {{TaskManagerOptions#NUM_TASK_SLOTS}}:
> {code:java}
> public static final ConfigOption MESOS_RM_TASKS_SLOTS =
>   key(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS)
>   .defaultValue(1);
> public static final ConfigOption NUM_TASK_SLOTS =
>   key("taskmanager.numberOfTaskSlots")
>   .defaultValue(1)
>   .withDescription("...");
> {code}
> This pattern is problematic as this creates 2 documentation entries for 
> {{taskmanager.numberOfTaskSlots}} with different descriptions, and opens the 
> potential for different defaults. Ultimately this causes the documentation to 
> become ambiguous.
> I thus propose to either outright remove this option or turn it into an 
> actual alias:
> {code:java}
> public static final ConfigOption MESOS_RM_TASKS_SLOTS =
> TaskManagerOptions.NUM_TASK_SLOTS;
> {code}
> As a side-effect of FLINK-8683 we can ensure that no differing config options 
> exist for a given key.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink issue #5519: [FLINK-8684] [config] Rework MesosTaskManagerParameters#M...

2018-02-18 Thread zentol

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5519
  
Correct, I've included a fix for FLINK-8684 in the 
[PR](https://github.com/apache/flink/pull/5515) for FLINK-8683.


---

[jira] [Commented] (FLINK-8274) Fix Java 64K method compiling limitation for CommonCalc

2018-02-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368465#comment-16368465
 ] 

ASF GitHub Bot commented on FLINK-8274:
---

Github user jahandarm commented on a diff in the pull request:

https://github.com/apache/flink/pull/5174#discussion_r168942867
  
--- Diff: 
flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/stream/table/CalcWithSplitCodeGenITCase.scala
 ---
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.stream.table
+
+import org.apache.flink.api.scala._
+import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
+import org.apache.flink.streaming.util.StreamingMultipleProgramsTestBase
--- End diff --

I think this class doesn't exist.


> Fix Java 64K method compiling limitation for CommonCalc
> ---
>
> Key: FLINK-8274
> URL: https://issues.apache.org/jira/browse/FLINK-8274
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API  SQL
>Affects Versions: 1.5.0
>Reporter: Ruidong Li
>Assignee: Ruidong Li
>Priority: Major
>
> For complex SQL Queries, the generated code for {code}DataStreamCalc{code}, 
> {code}DataSetCalc{code} may exceed Java's method length limitation 64kb.
>  
> This issue will split long method to several sub method calls.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] flink pull request #5174: [FLINK-8274][TableAPI & SQL] Fix Java 64K method c...

2018-02-18 Thread jahandarm

Github user jahandarm commented on a diff in the pull request:

https://github.com/apache/flink/pull/5174#discussion_r168942867
  
--- Diff: 
flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/stream/table/CalcWithSplitCodeGenITCase.scala
 ---
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.stream.table
+
+import org.apache.flink.api.scala._
+import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
+import org.apache.flink.streaming.util.StreamingMultipleProgramsTestBase
--- End diff --

I think this class doesn't exist.


---

67 matches

Mail list logo