[jira] [Commented] (FLINK-8811) Add MiniClusterClient to allow fast MiniCluster operations

2018-03-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383436#comment-16383436
 ] 

ASF GitHub Bot commented on FLINK-8811:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/5600#discussion_r171814808
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/minicluster/MiniCluster.java
 ---
@@ -178,6 +181,13 @@ public URI getRestAddress() {
}
}
 
+   public HighAvailabilityServices getHighAvailabilityServices() {
+   synchronized (lock) {
+   checkState(running, "MiniCluster is not yet running.");
--- End diff --

Because all state accesses to the MiniCluster are guarded since it can be 
used by potentially multiple threads.


> Add MiniClusterClient to allow fast MiniCluster operations
> --
>
> Key: FLINK-8811
> URL: https://issues.apache.org/jira/browse/FLINK-8811
> Project: Flink
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: flip-6
> Fix For: 1.5.0, 1.6.0
>
>
> We should offer a {{ClusterClient}} implementation for the {{MiniCluster}}. 
> That way we would be able to submit and wait for result without polling how 
> it would be the case by using the {{RestClusterClient}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8811) Add MiniClusterClient to allow fast MiniCluster operations

2018-03-01 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383313#comment-16383313
 ] 

ASF GitHub Bot commented on FLINK-8811:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5600


> Add MiniClusterClient to allow fast MiniCluster operations
> --
>
> Key: FLINK-8811
> URL: https://issues.apache.org/jira/browse/FLINK-8811
> Project: Flink
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: flip-6
> Fix For: 1.5.0, 1.6.0
>
>
> We should offer a {{ClusterClient}} implementation for the {{MiniCluster}}. 
> That way we would be able to submit and wait for result without polling how 
> it would be the case by using the {{RestClusterClient}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8811) Add MiniClusterClient to allow fast MiniCluster operations

2018-03-01 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383071#comment-16383071
 ] 

ASF GitHub Bot commented on FLINK-8811:
---

Github user zhangminglei commented on a diff in the pull request:

https://github.com/apache/flink/pull/5600#discussion_r171754724
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/minicluster/MiniCluster.java
 ---
@@ -178,6 +181,13 @@ public URI getRestAddress() {
}
}
 
+   public HighAvailabilityServices getHighAvailabilityServices() {
+   synchronized (lock) {
+   checkState(running, "MiniCluster is not yet running.");
--- End diff --

I am curious about the lock in this place. Is not such a scene as follow ?

Some tests or ITCase running in maven test simultaneously. And might be 
change this object at the same time. So, that is why we should have a lock here 
?


> Add MiniClusterClient to allow fast MiniCluster operations
> --
>
> Key: FLINK-8811
> URL: https://issues.apache.org/jira/browse/FLINK-8811
> Project: Flink
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: flip-6
> Fix For: 1.5.0, 1.6.0
>
>
> We should offer a {{ClusterClient}} implementation for the {{MiniCluster}}. 
> That way we would be able to submit and wait for result without polling how 
> it would be the case by using the {{RestClusterClient}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8811) Add MiniClusterClient to allow fast MiniCluster operations

2018-03-01 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381915#comment-16381915
 ] 

ASF GitHub Bot commented on FLINK-8811:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/5600
  
The error originates from the legacy code. Not sure what I could have 
changed there to be honest. Will see whether it passes locally or not.


> Add MiniClusterClient to allow fast MiniCluster operations
> --
>
> Key: FLINK-8811
> URL: https://issues.apache.org/jira/browse/FLINK-8811
> Project: Flink
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: flip-6
> Fix For: 1.5.0, 1.6.0
>
>
> We should offer a {{ClusterClient}} implementation for the {{MiniCluster}}. 
> That way we would be able to submit and wait for result without polling how 
> it would be the case by using the {{RestClusterClient}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8811) Add MiniClusterClient to allow fast MiniCluster operations

2018-03-01 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381771#comment-16381771
 ] 

ASF GitHub Bot commented on FLINK-8811:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5600
  
+1 for merging this ASAP  

The error message seems a bit disconcerting, though:
```
Finished 
org.apache.flink.test.checkpointing.HAIncrementalRocksDbBackendEventTimeWindowCheckpointingITCase#testSlidingTimeWindow.
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:891)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at 
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: 
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Not 
enough free slots available to run the job. You can decrease the operator 
parallelism or increase the number of slots per TaskManager in the 
configuration. Task to schedule: < Attempt #3 (Source: Custom Source (1/1)) @ 
(unassigned) - [SCHEDULED] > with groupID < bc764cd8ddf7a0cff126f51c16239658 > 
in sharing group < 7c540813226d407fbf248f58c16b372c >. Resources available to 
scheduler: Number of instances=2, total number of slots=4, available slots=0
at 
org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleTask(Scheduler.java:281)
at 
org.apache.flink.runtime.jobmanager.scheduler.Scheduler.allocateSlot(Scheduler.java:155)
at 
org.apache.flink.runtime.executiongraph.Execution.lambda$allocateAndAssignSlotForExecution$1(Execution.java:489)
at 
java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981)
at 
java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124)
at 
org.apache.flink.runtime.executiongraph.Execution.allocateAndAssignSlotForExecution(Execution.java:487)
at 
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.allocateResourcesForAll(ExecutionJobVertex.java:520)
at 
org.apache.flink.runtime.executiongraph.ExecutionGraph.scheduleEager(ExecutionGraph.java:957)
at 
org.apache.flink.runtime.executiongraph.ExecutionGraph.scheduleForExecution(ExecutionGraph.java:887)
at 
org.apache.flink.runtime.executiongraph.ExecutionGraph.restart(ExecutionGraph.java:1282)
at 
org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestartCallback.triggerFullRecovery(ExecutionGraphRestartCallback.java:59)
at 
org.apache.flink.runtime.executiongraph.restart.FixedDelayRestartStrategy$1.run(FixedDelayRestartStrategy.java:68)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```


> Add MiniClusterClient to allow fast MiniCluster operations
> --
>
> Key: FLINK-8811
> URL: https://issues.apache.org/jira/browse/FLINK-8811
> Project: Flink
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: flip-6
> Fix For: 1.5.0, 1.6.0
>
>
> We should offer a {{ClusterClient}} implementation for the {{MiniCluster}}. 
> That way we would be able to submit and wait

[jira] [Commented] (FLINK-8811) Add MiniClusterClient to allow fast MiniCluster operations

2018-02-28 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380689#comment-16380689
 ] 

ASF GitHub Bot commented on FLINK-8811:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/5600

[FLINK-8811] [flip6] Add initial implementation of the MiniClusterClient

## What is the purpose of the change

The MiniClusterClient directly talks to the MiniCluster avoiding polling
latencies of th RestClusterClient.

cc: @aljoscha 

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink miniClusterClient

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5600.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5600


commit 308603035b54c46b0b3c6967138352694c563628
Author: Till Rohrmann 
Date:   2018-02-28T13:07:12Z

[FLINK-8808] [flip6] Allow RestClusterClient to connect to local dispatcher

The RestClusterClient resolves a dispatcher address without an explicit host
to 'localhost'. That way we allow the RestClusterClient to talk to a 
Dispatcher
which runs in a local ActorSystem.

commit f52f37d26aae2ea57e3bf2f4b7e1e15a41a4529f
Author: Till Rohrmann 
Date:   2018-02-28T16:36:39Z

[FLINK-8811] [flip6] Add initial implementation of the MiniClusterClient

The MiniClusterClient directly talks to the MiniCluster avoiding polling
latencies of th RestClusterClient.

commit caa6357f18eb1527e59c567cb3628df556b78f39
Author: Till Rohrmann 
Date:   2018-02-28T16:49:29Z

[FLINK-8811] [flip6] Implement MiniClusterClient#getJobStatus

commit 1088117c54f93aa314e249108237c2d2c04762d9
Author: Till Rohrmann 
Date:   2018-02-28T16:54:06Z

[FLINK-8811] [flip6] Implement MiniClusterClient#cancel




> Add MiniClusterClient to allow fast MiniCluster operations
> --
>
> Key: FLINK-8811
> URL: https://issues.apache.org/jira/browse/FLINK-8811
> Project: Flink
>  Issue Type: Improvement
>  Components: Tests
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: flip-6
> Fix For: 1.5.0, 1.6.0
>
>
> We should offer a {{ClusterClient}} implementation for the {{MiniCluster}}. 
> That way we would be able to submit and wait for result without polling how 
> it would be the case by using the {{RestClusterClient}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)