[GitHub] flink pull request: [FLINK-2928][web-dashboard] Fix confusing job ...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1421#issuecomment-162331113
  
Layout is tricky. But the JobID is a very important field (for example to 
copy for CLI commands) and needs to stay.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043989#comment-15043989
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773625
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/broadcast/BroadcastVariableMaterialization.java
 ---
@@ -85,9 +85,9 @@ public void materializeVariable(MutableReader reader, 
TypeSerializerFactory Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1386#issuecomment-162334713
  
All in all, pretty good change. Makes many parts of the code nicer. Also 
good java docs and comments!

Comments were added inline. Nothing major, few details.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044006#comment-15044006
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1386#issuecomment-162334713
  
All in all, pretty good change. Makes many parts of the code nicer. Also 
good java docs and comments!

Comments were added inline. Nothing major, few details.


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3008) Failing/restarting streaming jobs are buggy in the webfrontend

[Ufuk Celebi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ufuk Celebi closed FLINK-3008.
--
   Resolution: Duplicate
Fix Version/s: 0.10.1

> Failing/restarting streaming jobs are buggy in the webfrontend
> --
>
> Key: FLINK-3008
> URL: https://issues.apache.org/jira/browse/FLINK-3008
> Project: Flink
>  Issue Type: Bug
>  Components: Command-line client, Webfrontend
>Affects Versions: 0.10.0, 1.0.0
>Reporter: Gyula Fora
> Fix For: 0.10.1
>
>
> Failing/restarting jobs are not handled very cleanely by the webfrontend. 
> There are multiple issues:
> - One cannot cancel a restarting job (there is no cancel button): this is a 
> problem if its a nonrecoverable failure and you just want to kill it
> - Sometimes restarting but manually canceled jobs stay in the running jobs 
> section in restarting phase even though it has been canceled. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1386#issuecomment-162330497
  
Checking this out...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3126) Remove accumulator type from "value" in web frontend

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043956#comment-15043956
 ] 

Stephan Ewen commented on FLINK-3126:
-

That is part of the "toString()"  method in many accumulators. We could either:

  - Remove that from the toString() methods
  - or use {{getLocalValue().toString()}} for what is displayed.

If possible, variant 2 seems better...

> Remove accumulator type from "value" in web frontend
> 
>
> Key: FLINK-3126
> URL: https://issues.apache.org/jira/browse/FLINK-3126
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Reporter: Robert Metzger
>
> The web frontend shows the type of the counter twice:
> http://i.imgur.com/yBWT8GR.png
> It would be nicer to just show the value "42" instead of "LongValue 42" there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043957#comment-15043957
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1386#issuecomment-162330497
  
Checking this out...


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-2765) Upgrade hbase version for hadoop-2 to 1.1 release

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-2765:
--
Description: 
Currently 0.98.11 is used:
{code}
0.98.11-hadoop2
{code}
Stable release for hadoop-2 is 1.1.x line
We should upgrade to 1.1.3 (soon to be released)

  was:
Currently 0.98.11 is used:
{code}
0.98.11-hadoop2
{code}

Stable release for hadoop-2 is 1.1.x line
We should upgrade to 1.1.3 (soon to be released)


> Upgrade hbase version for hadoop-2 to 1.1 release
> -
>
> Key: FLINK-2765
> URL: https://issues.apache.org/jira/browse/FLINK-2765
> Project: Flink
>  Issue Type: Improvement
>Reporter: Ted Yu
>Priority: Minor
>
> Currently 0.98.11 is used:
> {code}
> 0.98.11-hadoop2
> {code}
> Stable release for hadoop-2 is 1.1.x line
> We should upgrade to 1.1.3 (soon to be released)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043970#comment-15043970
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773363
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/functions/RuntimeContext.java
 ---
@@ -65,6 +65,22 @@
int getIndexOfThisSubtask();
 
/**
+* Gets the attempt number of this parallel subtask. First attempt is 
numbered 0.
--- End diff --

I would add here something like: "The attempt number corresponds to how 
many times the task has been restarted (after a failure or cancellation) since 
the job was initially started."


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773630
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/broadcast/BroadcastVariableMaterialization.java
 ---
@@ -178,9 +178,9 @@ private boolean decrementReferenceInternal(BatchTask referenceHolder, bool
if (errorIfNoReference) {
throw new IllegalStateException(
String.format("The task 
%s (%d/%d) did not hold a reference to the broadcast variable %s.",
-   
referenceHolder.getEnvironment().getTaskName(),
-   
referenceHolder.getEnvironment().getIndexInSubtaskGroup() + 1,
-   
referenceHolder.getEnvironment().getNumberOfSubtasks(),
+   
referenceHolder.getEnvironment().getTaskInfo().getTaskName(),
--- End diff --

Can replace this with `getTaskInfo().getTaskNameWithSubtasks()`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043990#comment-15043990
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773630
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/broadcast/BroadcastVariableMaterialization.java
 ---
@@ -178,9 +178,9 @@ private boolean decrementReferenceInternal(BatchTask referenceHolder, bool
if (errorIfNoReference) {
throw new IllegalStateException(
String.format("The task 
%s (%d/%d) did not hold a reference to the broadcast variable %s.",
-   
referenceHolder.getEnvironment().getTaskName(),
-   
referenceHolder.getEnvironment().getIndexInSubtaskGroup() + 1,
-   
referenceHolder.getEnvironment().getNumberOfSubtasks(),
+   
referenceHolder.getEnvironment().getTaskInfo().getTaskName(),
--- End diff --

Can replace this with `getTaskInfo().getTaskNameWithSubtasks()`.


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773625
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/broadcast/BroadcastVariableMaterialization.java
 ---
@@ -85,9 +85,9 @@ public void materializeVariable(MutableReader reader, 
TypeSerializerFactory

[GitHub] flink pull request: [FLINK-2978][web-dashboard][webclient] Integra...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1338#issuecomment-162334327
  
I've tested the change again on a cluster, works nicely!


![image](https://cloud.githubusercontent.com/assets/89049/11614821/5f2f4ca8-9c4e-11e5-87de-65ffa19f2b0e.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[sachingoel0101]

Github user sachingoel0101 commented on the pull request:

https://github.com/apache/flink/pull/1386#issuecomment-162325866
  
@StephanEwen can you take a look again? I have introduced the `TaskInfo` 
object in the second commit, and we can squash them before merging.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043937#comment-15043937
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user sachingoel0101 commented on the pull request:

https://github.com/apache/flink/pull/1386#issuecomment-162325866
  
@StephanEwen can you take a look again? I have introduced the `TaskInfo` 
object in the second commit, and we can squash them before merging.


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1518) Laggy task cancelling on task managers

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043952#comment-15043952
 ] 

Stephan Ewen commented on FLINK-1518:
-

@uce Can you double check this issue?

I don't think this has anything to do with the web frontend. Also, not sure if 
this is an issue in Flink altogether. How smooth cancellation works depends on 
whether the cancelled code cooperates fast, or takes long to clean up after 
itself.

> Laggy task cancelling on task managers
> --
>
> Key: FLINK-1518
> URL: https://issues.apache.org/jira/browse/FLINK-1518
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Runtime
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>
> When cancelling a task on a cluster setup via the web interface I've noticed 
> that task cancellation is laggy. The tasks switch to the CANCELING state, but 
> the cancelling of the tasks takes some time.
> There are also corresponding log messages:
> {code}
> 14:29:29,281 WARN  org.apache.flink.runtime.execution.RuntimeEnvironment  
>- Task CHAIN DataSource (at getTextDataSet(WordCount.java:141) 
> (org.apache.flink.api.java.io.TextInputF
> /02/11/2015 14:53:22CHAIN DataSource (at 
> getTextDataSet(WordCount.java:141) 
> (org.apache.flink.api.java.io.TextInputFormat)) -> FlatMap (FlatMap at 
> main(WordCount.java:67)) -> │ormat)) -> FlatMap (FlatMap at 
> main(WordCount.java:67)) -> Combine(SUM(1), at main(WordCount.java:67) (7/24) 
> did not react to cancelling signal. Sending repeated interrupt.
> {code}
> Tasks are cancelled at the task managers by repeatedly sending an interrupt 
> signal to the task threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1386#issuecomment-162331160
  
I am adding a number of comments. When you address them, please do not 
squash commits, so we can review just the changes separately.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2928) Confusing job status visualisation in web frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043961#comment-15043961
 ] 

ASF GitHub Bot commented on FLINK-2928:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1421#issuecomment-162331113
  
Layout is tricky. But the JobID is a very important field (for example to 
copy for CLI commands) and needs to stay.


> Confusing job status visualisation in web frontend
> --
>
> Key: FLINK-2928
> URL: https://issues.apache.org/jira/browse/FLINK-2928
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>Assignee: Sachin Goel
>Priority: Minor
>
> The web frontend displays the job status in very subtle way as a colored 
> circle next to the job name. For single tasks, the state is written out in 
> addition to the color coding (e.g. FAILED with a red background).
> I would like to add this for the job status as well.
> It can be confusing during restarts of a job to have single tasks marked as 
> "FAILED" w/o seeing easily what the job status is (RESTARTING).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043965#comment-15043965
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1386#issuecomment-162331160
  
I am adding a number of comments. When you address them, please do not 
squash commits, so we can review just the changes separately.


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773676
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/operators/BatchTask.java 
---
@@ -1150,8 +1149,8 @@ else if (index < 0 || index >= 
this.driver.getNumberOfDriverComparators()) {
 * @return The string for logging.
 */
public static String constructLogString(String message, String 
taskName, AbstractInvokable parent) {
-   return message + ":  " + taskName + " (" + 
(parent.getEnvironment().getIndexInSubtaskGroup() + 1) +
-   '/' + 
parent.getEnvironment().getNumberOfSubtasks() + ')';
+   return message + ":  " + taskName + " (" + 
(parent.getEnvironment().getTaskInfo().getIndexOfThisSubtask() + 1) +
--- End diff --

Can replace the subtask concatenation with 
`getTaskInfo().getTaskNameWithSubtasks()`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3023][web-dashboard] Display version an...

[rmetzger]

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/1422#discussion_r46773662
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/ClusterOverviewHandler.java
 ---
@@ -63,6 +67,10 @@ public String handleRequest(Map params, 
ActorGateway jobManager)
gen.writeNumberField("jobs-finished", 
overview.getNumJobsFinished());
gen.writeNumberField("jobs-cancelled", 
overview.getNumJobsCancelled());
gen.writeNumberField("jobs-failed", 
overview.getNumJobsFailed());
+   gen.writeStringField("flink-version", version);
+   if (commitID != null) {
--- End diff --

I think the commitID is `EnvironmentInformation.UNKNOWN` if its not 
available?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044003#comment-15044003
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-162334401
  
Tested the change on a cluster with a job reading some 100GBs of data ...
looks good ..

```
19:07:19,537 INFO  org.apache.flink.runtime.taskmanager.TaskManager 
 - Using 0.7 of the currently free heap space for Flink managed heap memory 
(9001 MB).
19:07:19,538 INFO  org.apache.flink.runtime.taskmanager.TaskManager 
 - MemoryManager will be initialized with lazy memory allocation.
```


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2978) Integrate web submission interface into the new dashboard

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044000#comment-15044000
 ] 

ASF GitHub Bot commented on FLINK-2978:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1338#issuecomment-162334327
  
I've tested the change again on a cluster, works nicely!


![image](https://cloud.githubusercontent.com/assets/89049/11614821/5f2f4ca8-9c4e-11e5-87de-65ffa19f2b0e.png)



> Integrate web submission interface into the new dashboard
> -
>
> Key: FLINK-2978
> URL: https://issues.apache.org/jira/browse/FLINK-2978
> Project: Flink
>  Issue Type: New Feature
>  Components: Web Client, Webfrontend
>Reporter: Sachin Goel
>Assignee: Sachin Goel
>
> As discussed in 
> http://mail-archives.apache.org/mod_mbox/flink-dev/201511.mbox/%3CCAL3J2zQg6UBKNDnm=8tshpz6r4p2jvx7nrlom7caajrb9s6...@mail.gmail.com%3E,
>  we should integrate job submission from the web into the dashboard.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3073] Replace Streaming Mode by Memory ...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-162334401
  
Tested the change on a cluster with a job reading some 100GBs of data ...
looks good ..

```
19:07:19,537 INFO  org.apache.flink.runtime.taskmanager.TaskManager 
 - Using 0.7 of the currently free heap space for Flink managed heap memory 
(9001 MB).
19:07:19,538 INFO  org.apache.flink.runtime.taskmanager.TaskManager 
 - MemoryManager will be initialized with lazy memory allocation.
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773342
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/TaskInfo.java ---
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Encapsulates task-specific information: name, index of subtask, 
parallelism and attempt number.
+ */
+public class TaskInfo {
+
+   private final String taskName;
+   private final String taskNameWithSubtasks;
+   private final int indexOfSubtask;
+   private final int numberOfParallelSubtasks;
+   private final int attemptNumber;
+
+   public TaskInfo(String taskName, int indexOfSubtask, int 
numberOfParallelSubtasks, int attemptNumber) {
+   Preconditions.checkArgument(indexOfSubtask >= 0, "Task index 
must be a non-negative number.");
+   Preconditions.checkArgument(numberOfParallelSubtasks >= 1, 
"Parallelism must be a positive number.");
+   Preconditions.checkArgument(indexOfSubtask < 
numberOfParallelSubtasks, "Task index must be less than parallelism.");
+   Preconditions.checkArgument(attemptNumber >= 0, "Attempt number 
must be a non-negative number.");
--- End diff --

Good set of checks. Would be good to add a non-null check for `taskName` as 
well.
Optional: statically importing ´checkArgument` makes it even nicer.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043967#comment-15043967
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773342
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/TaskInfo.java ---
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Encapsulates task-specific information: name, index of subtask, 
parallelism and attempt number.
+ */
+public class TaskInfo {
+
+   private final String taskName;
+   private final String taskNameWithSubtasks;
+   private final int indexOfSubtask;
+   private final int numberOfParallelSubtasks;
+   private final int attemptNumber;
+
+   public TaskInfo(String taskName, int indexOfSubtask, int 
numberOfParallelSubtasks, int attemptNumber) {
+   Preconditions.checkArgument(indexOfSubtask >= 0, "Task index 
must be a non-negative number.");
+   Preconditions.checkArgument(numberOfParallelSubtasks >= 1, 
"Parallelism must be a positive number.");
+   Preconditions.checkArgument(indexOfSubtask < 
numberOfParallelSubtasks, "Task index must be less than parallelism.");
+   Preconditions.checkArgument(attemptNumber >= 0, "Attempt number 
must be a non-negative number.");
--- End diff --

Good set of checks. Would be good to add a non-null check for `taskName` as 
well.
Optional: statically importing ´checkArgument` makes it even nicer.


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043993#comment-15043993
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773676
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/operators/BatchTask.java 
---
@@ -1150,8 +1149,8 @@ else if (index < 0 || index >= 
this.driver.getNumberOfDriverComparators()) {
 * @return The string for logging.
 */
public static String constructLogString(String message, String 
taskName, AbstractInvokable parent) {
-   return message + ":  " + taskName + " (" + 
(parent.getEnvironment().getIndexInSubtaskGroup() + 1) +
-   '/' + 
parent.getEnvironment().getNumberOfSubtasks() + ')';
+   return message + ":  " + taskName + " (" + 
(parent.getEnvironment().getTaskInfo().getIndexOfThisSubtask() + 1) +
--- End diff --

Can replace the subtask concatenation with 
`getTaskInfo().getTaskNameWithSubtasks()`.


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773689
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java ---
@@ -240,14 +234,13 @@ public Task(TaskDeploymentDescriptor tdd,
checkArgument(tdd.getNumberOfSubtasks() > 0);
checkArgument(tdd.getIndexInSubtaskGroup() >= 0);
checkArgument(tdd.getIndexInSubtaskGroup() < 
tdd.getNumberOfSubtasks());
+   checkArgument(tdd.getAttemptNumber() >= 0);
--- End diff --

You can probably skip these checks, given that `TaskInfo` has these checks 
already...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043996#comment-15043996
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773689
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java ---
@@ -240,14 +234,13 @@ public Task(TaskDeploymentDescriptor tdd,
checkArgument(tdd.getNumberOfSubtasks() > 0);
checkArgument(tdd.getIndexInSubtaskGroup() >= 0);
checkArgument(tdd.getIndexInSubtaskGroup() < 
tdd.getNumberOfSubtasks());
+   checkArgument(tdd.getAttemptNumber() >= 0);
--- End diff --

You can probably skip these checks, given that `TaskInfo` has these checks 
already...


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1518) Laggy task cancelling on task managers

[Ufuk Celebi (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044036#comment-15044036
 ] 

Ufuk Celebi commented on FLINK-1518:


Yes, will do.

Sorry for the confusing formulation. I didn't mean to say that this is related 
to the web interface. Just noticed it there.

> Laggy task cancelling on task managers
> --
>
> Key: FLINK-1518
> URL: https://issues.apache.org/jira/browse/FLINK-1518
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Runtime
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>
> When cancelling a task on a cluster setup via the web interface I've noticed 
> that task cancellation is laggy. The tasks switch to the CANCELING state, but 
> the cancelling of the tasks takes some time.
> There are also corresponding log messages:
> {code}
> 14:29:29,281 WARN  org.apache.flink.runtime.execution.RuntimeEnvironment  
>- Task CHAIN DataSource (at getTextDataSet(WordCount.java:141) 
> (org.apache.flink.api.java.io.TextInputF
> /02/11/2015 14:53:22CHAIN DataSource (at 
> getTextDataSet(WordCount.java:141) 
> (org.apache.flink.api.java.io.TextInputFormat)) -> FlatMap (FlatMap at 
> main(WordCount.java:67)) -> │ormat)) -> FlatMap (FlatMap at 
> main(WordCount.java:67)) -> Combine(SUM(1), at main(WordCount.java:67) (7/24) 
> did not react to cancelling signal. Sending repeated interrupt.
> {code}
> Tasks are cancelled at the task managers by repeatedly sending an interrupt 
> signal to the task threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3055] remove getSubTaskIndex()

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1438#issuecomment-162330308
  
Thanks, looks good.

Will merge this...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3055) ExecutionVertex has duplicate method getParallelSubtaskIndex and getSubTaskIndex

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043954#comment-15043954
 ] 

ASF GitHub Bot commented on FLINK-3055:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1438#issuecomment-162330308
  
Thanks, looks good.

Will merge this...


> ExecutionVertex has duplicate method getParallelSubtaskIndex and 
> getSubTaskIndex
> 
>
> Key: FLINK-3055
> URL: https://issues.apache.org/jira/browse/FLINK-3055
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Runtime
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>Assignee: jun aoki
>Priority: Trivial
>
> In {{ExecutionVertex}}:
> {code}
> public int getSubTaskIndex() {
>   return subTaskIndex;
> }
> public int getParallelSubtaskIndex() {
>   return this.subTaskIndex;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3023) Show Flink version + commit id for -SNAPSHOT versions in web frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043991#comment-15043991
 ] 

ASF GitHub Bot commented on FLINK-3023:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1422#issuecomment-162333939
  
Tested the change on a cluster

![image](https://cloud.githubusercontent.com/assets/89049/11614772/3c8c2ec4-9c4d-11e5-870e-0d75586d53ae.png)



> Show Flink version + commit id for -SNAPSHOT versions in web frontend
> -
>
> Key: FLINK-3023
> URL: https://issues.apache.org/jira/browse/FLINK-3023
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>
> The old frontend was showing the Flink version and the commit id for SNAPSHOT 
> builds.
> This is a helpful feature to quickly see which Flink version is running.
> It would be nice to add this again to the web interface.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3023][web-dashboard] Display version an...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1422#issuecomment-162333939
  
Tested the change on a cluster

![image](https://cloud.githubusercontent.com/assets/89049/11614772/3c8c2ec4-9c4d-11e5-870e-0d75586d53ae.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044007#comment-15044007
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-162334879
  
As followup to this, I would suggest two changes:

1. Adjust the log message, to reflect that not 0.7 are either used 
immediately (eager allocation) or that the managed memory is capped at 0.7 * 
total memory. That makes it clear from reading the log that the memory is not 
pre-allocated.

2. The web frontend should display for managed memory: (used / max). I have 
talked to users who were confused, thinking despite starting the system in 
streaming mode, it pre-allocated all the heap way, 


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3073] Replace Streaming Mode by Memory ...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-162334879
  
As followup to this, I would suggest two changes:

1. Adjust the log message, to reflect that not 0.7 are either used 
immediately (eager allocation) or that the managed memory is capped at 0.7 * 
total memory. That makes it clear from reading the log that the memory is not 
pre-allocated.

2. The web frontend should display for managed memory: (used / max). I have 
talked to users who were confused, thinking despite starting the system in 
streaming mode, it pre-allocated all the heap way, 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3125) Web dashboard does not start when log files are not found

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-3125:
---

 Summary: Web dashboard does not start when log files are not found
 Key: FLINK-3125
 URL: https://issues.apache.org/jira/browse/FLINK-3125
 Project: Flink
  Issue Type: Bug
  Components: Webfrontend
Affects Versions: 0.10.1
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 0.7.1-incubating, 1.0.0, 0.10.2






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773363
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/api/common/functions/RuntimeContext.java
 ---
@@ -65,6 +65,22 @@
int getIndexOfThisSubtask();
 
/**
+* Gets the attempt number of this parallel subtask. First attempt is 
numbered 0.
--- End diff --

I would add here something like: "The attempt number corresponds to how 
many times the task has been restarted (after a failure or cancellation) since 
the job was initially started."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043997#comment-15043997
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773693
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java ---
@@ -120,16 +121,9 @@
/** The execution attempt of the parallel subtask */
private final ExecutionAttemptID executionId;
 
-   /** The index of the parallel subtask, in [0, numberOfSubtasks) */
-   private final int subtaskIndex;
+   /** TaskInfo object for this task */
+   private final TaskInfo taskInfo;
 
-   /** The number of parallel subtasks for the 
JobVertex/ExecutionJobVertex that this task belongs to */
-   private final int parallelism;
-
-   /** The name of the task */
-   private final String taskName;
-
-   /** The name of the task, including the subtask index and the 
parallelism */
private final String taskNameWithSubtask;
--- End diff --

This field is also subsumed by the `TaskInfo`


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773693
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java ---
@@ -120,16 +121,9 @@
/** The execution attempt of the parallel subtask */
private final ExecutionAttemptID executionId;
 
-   /** The index of the parallel subtask, in [0, numberOfSubtasks) */
-   private final int subtaskIndex;
+   /** TaskInfo object for this task */
+   private final TaskInfo taskInfo;
 
-   /** The number of parallel subtasks for the 
JobVertex/ExecutionJobVertex that this task belongs to */
-   private final int parallelism;
-
-   /** The name of the task */
-   private final String taskName;
-
-   /** The name of the task, including the subtask index and the 
parallelism */
private final String taskNameWithSubtask;
--- End diff --

This field is also subsumed by the `TaskInfo`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773807
  
--- Diff: 
flink-tests/src/test/java/org/apache/flink/test/misc/AttemptNumberITCase.java 
---
@@ -0,0 +1,70 @@
+/*
--- End diff --

In order to reduce the number of redundant (but time consuming) tests: Can 
you fuse this test with the TaskFailureITCase?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044005#comment-15044005
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46773807
  
--- Diff: 
flink-tests/src/test/java/org/apache/flink/test/misc/AttemptNumberITCase.java 
---
@@ -0,0 +1,70 @@
+/*
--- End diff --

In order to reduce the number of redundant (but time consuming) tests: Can 
you fuse this test with the TaskFailureITCase?


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: JobManager Data Structure doc fix

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1430#issuecomment-162335117
  
Merging this...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3023) Show Flink version + commit id for -SNAPSHOT versions in web frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044083#comment-15044083
 ] 

ASF GitHub Bot commented on FLINK-3023:
---

Github user sachingoel0101 commented on a diff in the pull request:

https://github.com/apache/flink/pull/1422#discussion_r46774718
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/ClusterOverviewHandler.java
 ---
@@ -63,6 +67,10 @@ public String handleRequest(Map params, 
ActorGateway jobManager)
gen.writeNumberField("jobs-finished", 
overview.getNumJobsFinished());
gen.writeNumberField("jobs-cancelled", 
overview.getNumJobsCancelled());
gen.writeNumberField("jobs-failed", 
overview.getNumJobsFailed());
+   gen.writeStringField("flink-version", version);
+   if (commitID != null) {
--- End diff --

Gah. Apologies. I should've been more careful. Fixed it. 


> Show Flink version + commit id for -SNAPSHOT versions in web frontend
> -
>
> Key: FLINK-3023
> URL: https://issues.apache.org/jira/browse/FLINK-3023
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>
> The old frontend was showing the Flink version and the commit id for SNAPSHOT 
> builds.
> This is a helpful feature to quickly see which Flink version is running.
> It would be nice to add this again to the web interface.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3125) Web dashboard does not start when log files are not found

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-3125.
---

> Web dashboard does not start when log files are not found
> -
>
> Key: FLINK-3125
> URL: https://issues.apache.org/jira/browse/FLINK-3125
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 0.10.1
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.0.0, 0.10.2
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-3125) Web dashboard does not start when log files are not found

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-3125.
-
   Resolution: Fixed
Fix Version/s: (was: 0.7.1-incubating)

Fixed in 
  - 1.0.0 via a1f4dd1f221e6d3eb0e9045350faa29fc9e6d2e1
  - 0.10.2 via a368221006033ae8a48d2e81dc535a3eda0b0b87

> Web dashboard does not start when log files are not found
> -
>
> Key: FLINK-3125
> URL: https://issues.apache.org/jira/browse/FLINK-3125
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 0.10.1
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.0.0, 0.10.2
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[sachingoel0101]

Github user sachingoel0101 commented on the pull request:

https://github.com/apache/flink/pull/1386#issuecomment-162345554
  
@StephanEwen thanks for the review. I have addressed all but one of your 
comments. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: JobManager Data Structure doc fix

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1430


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3055] remove getSubTaskIndex()

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1438


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2978][web-dashboard][webclient] Integra...

[sachingoel0101]

Github user sachingoel0101 commented on the pull request:

https://github.com/apache/flink/pull/1338#issuecomment-162340095
  
@rmetzger thanks. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-2524) Add "getTaskNameWithSubtasks()" to RuntimeContext

[Sachin Goel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sachin Goel reassigned FLINK-2524:
--

Assignee: Sachin Goel

> Add "getTaskNameWithSubtasks()" to RuntimeContext
> -
>
> Key: FLINK-2524
> URL: https://issues.apache.org/jira/browse/FLINK-2524
> Project: Flink
>  Issue Type: Improvement
>  Components: Local Runtime
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Sachin Goel
>  Labels: easyfix, starter
> Fix For: 1.0.0
>
>
> When printing information to logs or debug output, one frequently needs to 
> identify the statement with the originating task (task name and which 
> subtask).
> In many places, the system and user code manually construct something like 
> "MyTask (2/7)".
> The {{RuntimeContext}} should offer this, because it is too frequently needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1731) Add kMeans clustering algorithm to machine learning library

[Ovidiu Marcu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044136#comment-15044136
 ] 

Ovidiu Marcu commented on FLINK-1731:
-

Will you consider this issue within the next release?

> Add kMeans clustering algorithm to machine learning library
> ---
>
> Key: FLINK-1731
> URL: https://issues.apache.org/jira/browse/FLINK-1731
> Project: Flink
>  Issue Type: New Feature
>  Components: Machine Learning Library
>Reporter: Till Rohrmann
>Assignee: Peter Schrott
>  Labels: ML
>
> The Flink repository already contains a kMeans implementation but it is not 
> yet ported to the machine learning library. I assume that only the used data 
> types have to be adapted and then it can be more or less directly moved to 
> flink-ml.
> The kMeans++ [1] and the kMeans|| [2] algorithm constitute a better 
> implementation because the improve the initial seeding phase to achieve near 
> optimal clustering. It might be worthwhile to implement kMeans||.
> Resources:
> [1] http://ilpubs.stanford.edu:8090/778/1/2006-13.pdf
> [2] http://theory.stanford.edu/~sergei/papers/vldb12-kmpar.pdf



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044122#comment-15044122
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user sachingoel0101 commented on the pull request:

https://github.com/apache/flink/pull/1386#issuecomment-162345554
  
@StephanEwen thanks for the review. I have addressed all but one of your 
comments. 


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044121#comment-15044121
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user sachingoel0101 commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46775517
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/operators/BatchTask.java 
---
@@ -1150,8 +1149,8 @@ else if (index < 0 || index >= 
this.driver.getNumberOfDriverComparators()) {
 * @return The string for logging.
 */
public static String constructLogString(String message, String 
taskName, AbstractInvokable parent) {
-   return message + ":  " + taskName + " (" + 
(parent.getEnvironment().getIndexInSubtaskGroup() + 1) +
-   '/' + 
parent.getEnvironment().getNumberOfSubtasks() + ')';
+   return message + ":  " + taskName + " (" + 
(parent.getEnvironment().getTaskInfo().getIndexOfThisSubtask() + 1) +
--- End diff --

That can lead to problems in handling of Chained Tasks. For those, the name 
of chained task, but the index and parallelism of parent is used.


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[sachingoel0101]

Github user sachingoel0101 commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46775517
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/operators/BatchTask.java 
---
@@ -1150,8 +1149,8 @@ else if (index < 0 || index >= 
this.driver.getNumberOfDriverComparators()) {
 * @return The string for logging.
 */
public static String constructLogString(String message, String 
taskName, AbstractInvokable parent) {
-   return message + ":  " + taskName + " (" + 
(parent.getEnvironment().getIndexInSubtaskGroup() + 1) +
-   '/' + 
parent.getEnvironment().getNumberOfSubtasks() + ')';
+   return message + ":  " + taskName + " (" + 
(parent.getEnvironment().getTaskInfo().getIndexOfThisSubtask() + 1) +
--- End diff --

That can lead to problems in handling of Chained Tasks. For those, the name 
of chained task, but the index and parallelism of parent is used.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2978][web-dashboard][webclient] Integra...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1338#issuecomment-162349944
  
@sachingoel0101 , This looks good, let's get it merged soon.
Some comments on your questions:

1) Concerning the config key: Good that way, thanks!

2) Concerning path parameters vs query parameters: I am not an expert on 
REST, but I think they serve different roles, so let's keep them separate. I 
think that two maps are actually not bad. Makes the distinction obvious and is 
efficient. I am not so convinced of the enum-based approach...

3) Concerning error handling: Let's leave it as it is in this pull request 
and consolidate it in a separate effort. Should not block this PR:


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2978) Integrate web submission interface into the new dashboard

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044158#comment-15044158
 ] 

ASF GitHub Bot commented on FLINK-2978:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1338#issuecomment-162349944
  
@sachingoel0101 , This looks good, let's get it merged soon.
Some comments on your questions:

1) Concerning the config key: Good that way, thanks!

2) Concerning path parameters vs query parameters: I am not an expert on 
REST, but I think they serve different roles, so let's keep them separate. I 
think that two maps are actually not bad. Makes the distinction obvious and is 
efficient. I am not so convinced of the enum-based approach...

3) Concerning error handling: Let's leave it as it is in this pull request 
and consolidate it in a separate effort. Should not block this PR:


> Integrate web submission interface into the new dashboard
> -
>
> Key: FLINK-2978
> URL: https://issues.apache.org/jira/browse/FLINK-2978
> Project: Flink
>  Issue Type: New Feature
>  Components: Web Client, Webfrontend
>Reporter: Sachin Goel
>Assignee: Sachin Goel
>
> As discussed in 
> http://mail-archives.apache.org/mod_mbox/flink-dev/201511.mbox/%3CCAL3J2zQg6UBKNDnm=8tshpz6r4p2jvx7nrlom7caajrb9s6...@mail.gmail.com%3E,
>  we should integrate job submission from the web into the dashboard.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3023][web-dashboard] Display version an...

[sachingoel0101]

Github user sachingoel0101 commented on a diff in the pull request:

https://github.com/apache/flink/pull/1422#discussion_r46774718
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/ClusterOverviewHandler.java
 ---
@@ -63,6 +67,10 @@ public String handleRequest(Map params, 
ActorGateway jobManager)
gen.writeNumberField("jobs-finished", 
overview.getNumJobsFinished());
gen.writeNumberField("jobs-cancelled", 
overview.getNumJobsCancelled());
gen.writeNumberField("jobs-failed", 
overview.getNumJobsFailed());
+   gen.writeStringField("flink-version", version);
+   if (commitID != null) {
--- End diff --

Gah. Apologies. I should've been more careful. Fixed it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2928][web-dashboard] Fix confusing job ...

[sachingoel0101]

Github user sachingoel0101 commented on the pull request:

https://github.com/apache/flink/pull/1421#issuecomment-162341452
  
Makes sense. Lemme figure something out here, and I will post an update in 
a day or two.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2928) Confusing job status visualisation in web frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044093#comment-15044093
 ] 

ASF GitHub Bot commented on FLINK-2928:
---

Github user sachingoel0101 commented on the pull request:

https://github.com/apache/flink/pull/1421#issuecomment-162341452
  
Makes sense. Lemme figure something out here, and I will post an update in 
a day or two.


> Confusing job status visualisation in web frontend
> --
>
> Key: FLINK-2928
> URL: https://issues.apache.org/jira/browse/FLINK-2928
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>Assignee: Sachin Goel
>Priority: Minor
>
> The web frontend displays the job status in very subtle way as a colored 
> circle next to the job name. For single tasks, the state is written out in 
> addition to the color coding (e.g. FAILED with a red background).
> I would like to add this for the job status as well.
> It can be confusing during restarts of a job to have single tasks marked as 
> "FAILED" w/o seeing easily what the job status is (RESTARTING).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2488) Expose attemptNumber in RuntimeContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044152#comment-15044152
 ] 

ASF GitHub Bot commented on FLINK-2488:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46776269
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/operators/BatchTask.java 
---
@@ -1150,8 +1149,8 @@ else if (index < 0 || index >= 
this.driver.getNumberOfDriverComparators()) {
 * @return The string for logging.
 */
public static String constructLogString(String message, String 
taskName, AbstractInvokable parent) {
-   return message + ":  " + taskName + " (" + 
(parent.getEnvironment().getIndexInSubtaskGroup() + 1) +
-   '/' + 
parent.getEnvironment().getNumberOfSubtasks() + ')';
+   return message + ":  " + taskName + " (" + 
(parent.getEnvironment().getTaskInfo().getIndexOfThisSubtask() + 1) +
--- End diff --

Ah, I see...


> Expose attemptNumber in RuntimeContext
> --
>
> Key: FLINK-2488
> URL: https://issues.apache.org/jira/browse/FLINK-2488
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager, TaskManager
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Sachin Goel
>Priority: Minor
>
> It would be nice to expose the attemptNumber of a task in the 
> {{RuntimeContext}}. 
> This would allow user code to behave differently in restart scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2488] Expose Attempt Number in RuntimeC...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1386#discussion_r46776269
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/operators/BatchTask.java 
---
@@ -1150,8 +1149,8 @@ else if (index < 0 || index >= 
this.driver.getNumberOfDriverComparators()) {
 * @return The string for logging.
 */
public static String constructLogString(String message, String 
taskName, AbstractInvokable parent) {
-   return message + ":  " + taskName + " (" + 
(parent.getEnvironment().getIndexInSubtaskGroup() + 1) +
-   '/' + 
parent.getEnvironment().getNumberOfSubtasks() + ')';
+   return message + ":  " + taskName + " (" + 
(parent.getEnvironment().getTaskInfo().getIndexOfThisSubtask() + 1) +
--- End diff --

Ah, I see...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-3055) ExecutionVertex has duplicate method getParallelSubtaskIndex and getSubTaskIndex

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-3055.
-
   Resolution: Fixed
Fix Version/s: 1.0.0

Fixed via 4003352cdbb7cabf9446d73412994c3ab731e9c0

Thank you for the contribution!

> ExecutionVertex has duplicate method getParallelSubtaskIndex and 
> getSubTaskIndex
> 
>
> Key: FLINK-3055
> URL: https://issues.apache.org/jira/browse/FLINK-3055
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Runtime
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>Assignee: jun aoki
>Priority: Trivial
> Fix For: 1.0.0
>
>
> In {{ExecutionVertex}}:
> {code}
> public int getSubTaskIndex() {
>   return subTaskIndex;
> }
> public int getParallelSubtaskIndex() {
>   return this.subTaskIndex;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3055) ExecutionVertex has duplicate method getParallelSubtaskIndex and getSubTaskIndex

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-3055.
---

> ExecutionVertex has duplicate method getParallelSubtaskIndex and 
> getSubTaskIndex
> 
>
> Key: FLINK-3055
> URL: https://issues.apache.org/jira/browse/FLINK-3055
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Runtime
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>Assignee: jun aoki
>Priority: Trivial
> Fix For: 1.0.0
>
>
> In {{ExecutionVertex}}:
> {code}
> public int getSubTaskIndex() {
>   return subTaskIndex;
> }
> public int getParallelSubtaskIndex() {
>   return this.subTaskIndex;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-7) [GitHub] Enable Range Partitioner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044496#comment-15044496
 ] 

ASF GitHub Bot commented on FLINK-7:


Github user ChengXiangLi commented on a diff in the pull request:

https://github.com/apache/flink/pull/1255#discussion_r46792344
  
--- Diff: 
flink-optimizer/src/main/java/org/apache/flink/optimizer/traversals/RangePartitionRewriter.java
 ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.optimizer.traversals;
+
+import org.apache.flink.api.common.distributions.CommonRangeBoundaries;
+import org.apache.flink.api.common.operators.UnaryOperatorInformation;
+import org.apache.flink.api.common.operators.base.GroupReduceOperatorBase;
+import org.apache.flink.api.common.operators.base.MapOperatorBase;
+import org.apache.flink.api.common.operators.base.MapPartitionOperatorBase;
+import org.apache.flink.api.common.operators.util.FieldList;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeComparatorFactory;
+import org.apache.flink.api.java.typeutils.RecordTypeInfo;
+import org.apache.flink.runtime.io.network.DataExchangeMode;
+import org.apache.flink.runtime.operators.udf.AssignRangeIndex;
+import org.apache.flink.runtime.operators.udf.PartitionIDRemoveWrapper;
+import org.apache.flink.runtime.operators.udf.RangeBoundaryBuilder;
+import org.apache.flink.api.java.functions.SampleInCoordinator;
+import org.apache.flink.api.java.functions.SampleInPartition;
+import org.apache.flink.api.java.sampling.IntermediateSampleData;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.optimizer.dag.GroupReduceNode;
+import org.apache.flink.optimizer.dag.MapNode;
+import org.apache.flink.optimizer.dag.MapPartitionNode;
+import org.apache.flink.optimizer.dag.TempMode;
+import org.apache.flink.optimizer.plan.Channel;
+import org.apache.flink.optimizer.plan.NamedChannel;
+import org.apache.flink.optimizer.plan.OptimizedPlan;
+import org.apache.flink.optimizer.plan.PlanNode;
+import org.apache.flink.optimizer.plan.SingleInputPlanNode;
+import org.apache.flink.optimizer.util.Utils;
+import org.apache.flink.runtime.operators.DriverStrategy;
+import org.apache.flink.runtime.operators.shipping.ShipStrategyType;
+import org.apache.flink.util.Visitor;
+
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+
+public class RangePartitionRewriter implements Visitor {
+   final static long SEED = org.apache.flink.api.java.Utils.RNG.nextLong();
+
+   final OptimizedPlan plan;
+
+   public RangePartitionRewriter(OptimizedPlan plan) {
+   this.plan = plan;
+   }
+
+   @Override
+   public boolean preVisit(PlanNode visitable) {
+   return true;
+   }
+
+   @Override
+   public void postVisit(PlanNode visitable) {
+   final List outgoingChannels = 
visitable.getOutgoingChannels();
+   final List newOutGoingChannels = new LinkedList<>();
+   final List toBeRemoveChannels = new ArrayList<>();
+   for (Channel channel : outgoingChannels) {
+   ShipStrategyType shipStrategy = 
channel.getShipStrategy();
+   if (shipStrategy == ShipStrategyType.PARTITION_RANGE) {
+   TypeInformation outputType = 
channel.getSource().getProgramOperator().getOperatorInfo().getOutputType();
+   // Do not optimize for record type, it's a 
special case for range partitioner, and should be removed later.
+   if (!(outputType instanceof RecordTypeInfo)) {
--- End diff --

[GitHub] flink pull request: [FLINK-7] [Runtime] Enable Range Partitioner.

[ChengXiangLi]

Github user ChengXiangLi commented on a diff in the pull request:

https://github.com/apache/flink/pull/1255#discussion_r46792344
  
--- Diff: 
flink-optimizer/src/main/java/org/apache/flink/optimizer/traversals/RangePartitionRewriter.java
 ---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.optimizer.traversals;
+
+import org.apache.flink.api.common.distributions.CommonRangeBoundaries;
+import org.apache.flink.api.common.operators.UnaryOperatorInformation;
+import org.apache.flink.api.common.operators.base.GroupReduceOperatorBase;
+import org.apache.flink.api.common.operators.base.MapOperatorBase;
+import org.apache.flink.api.common.operators.base.MapPartitionOperatorBase;
+import org.apache.flink.api.common.operators.util.FieldList;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeComparatorFactory;
+import org.apache.flink.api.java.typeutils.RecordTypeInfo;
+import org.apache.flink.runtime.io.network.DataExchangeMode;
+import org.apache.flink.runtime.operators.udf.AssignRangeIndex;
+import org.apache.flink.runtime.operators.udf.PartitionIDRemoveWrapper;
+import org.apache.flink.runtime.operators.udf.RangeBoundaryBuilder;
+import org.apache.flink.api.java.functions.SampleInCoordinator;
+import org.apache.flink.api.java.functions.SampleInPartition;
+import org.apache.flink.api.java.sampling.IntermediateSampleData;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.optimizer.dag.GroupReduceNode;
+import org.apache.flink.optimizer.dag.MapNode;
+import org.apache.flink.optimizer.dag.MapPartitionNode;
+import org.apache.flink.optimizer.dag.TempMode;
+import org.apache.flink.optimizer.plan.Channel;
+import org.apache.flink.optimizer.plan.NamedChannel;
+import org.apache.flink.optimizer.plan.OptimizedPlan;
+import org.apache.flink.optimizer.plan.PlanNode;
+import org.apache.flink.optimizer.plan.SingleInputPlanNode;
+import org.apache.flink.optimizer.util.Utils;
+import org.apache.flink.runtime.operators.DriverStrategy;
+import org.apache.flink.runtime.operators.shipping.ShipStrategyType;
+import org.apache.flink.util.Visitor;
+
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+
+public class RangePartitionRewriter implements Visitor {
+   final static long SEED = org.apache.flink.api.java.Utils.RNG.nextLong();
+
+   final OptimizedPlan plan;
+
+   public RangePartitionRewriter(OptimizedPlan plan) {
+   this.plan = plan;
+   }
+
+   @Override
+   public boolean preVisit(PlanNode visitable) {
+   return true;
+   }
+
+   @Override
+   public void postVisit(PlanNode visitable) {
+   final List outgoingChannels = 
visitable.getOutgoingChannels();
+   final List newOutGoingChannels = new LinkedList<>();
+   final List toBeRemoveChannels = new ArrayList<>();
+   for (Channel channel : outgoingChannels) {
+   ShipStrategyType shipStrategy = 
channel.getShipStrategy();
+   if (shipStrategy == ShipStrategyType.PARTITION_RANGE) {
+   TypeInformation outputType = 
channel.getSource().getProgramOperator().getOperatorInfo().getOutputType();
+   // Do not optimize for record type, it's a 
special case for range partitioner, and should be removed later.
+   if (!(outputType instanceof RecordTypeInfo)) {
--- End diff --

Thanks for the remind, @fhueske , I've added verification on 
`PartitionOperator` level and related unit test. BTW, would the `Record` type 
be fully discarded finally? it seems been broken on many features, but there