[jira] [Commented] (FLINK-8811) Add MiniClusterClient to allow fast MiniCluster operations
[ https://issues.apache.org/jira/browse/FLINK-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383436#comment-16383436 ] ASF GitHub Bot commented on FLINK-8811: --- Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5600#discussion_r171814808 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/minicluster/MiniCluster.java --- @@ -178,6 +181,13 @@ public URI getRestAddress() { } } + public HighAvailabilityServices getHighAvailabilityServices() { + synchronized (lock) { + checkState(running, "MiniCluster is not yet running."); --- End diff -- Because all state accesses to the MiniCluster are guarded since it can be used by potentially multiple threads. > Add MiniClusterClient to allow fast MiniCluster operations > -- > > Key: FLINK-8811 > URL: https://issues.apache.org/jira/browse/FLINK-8811 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.5.0, 1.6.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > Labels: flip-6 > Fix For: 1.5.0, 1.6.0 > > > We should offer a {{ClusterClient}} implementation for the {{MiniCluster}}. > That way we would be able to submit and wait for result without polling how > it would be the case by using the {{RestClusterClient}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8811) Add MiniClusterClient to allow fast MiniCluster operations
[ https://issues.apache.org/jira/browse/FLINK-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383313#comment-16383313 ] ASF GitHub Bot commented on FLINK-8811: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5600 > Add MiniClusterClient to allow fast MiniCluster operations > -- > > Key: FLINK-8811 > URL: https://issues.apache.org/jira/browse/FLINK-8811 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.5.0, 1.6.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > Labels: flip-6 > Fix For: 1.5.0, 1.6.0 > > > We should offer a {{ClusterClient}} implementation for the {{MiniCluster}}. > That way we would be able to submit and wait for result without polling how > it would be the case by using the {{RestClusterClient}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8811) Add MiniClusterClient to allow fast MiniCluster operations
[ https://issues.apache.org/jira/browse/FLINK-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383071#comment-16383071 ] ASF GitHub Bot commented on FLINK-8811: --- Github user zhangminglei commented on a diff in the pull request: https://github.com/apache/flink/pull/5600#discussion_r171754724 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/minicluster/MiniCluster.java --- @@ -178,6 +181,13 @@ public URI getRestAddress() { } } + public HighAvailabilityServices getHighAvailabilityServices() { + synchronized (lock) { + checkState(running, "MiniCluster is not yet running."); --- End diff -- I am curious about the lock in this place. Is not such a scene as follow ? Some tests or ITCase running in maven test simultaneously. And might be change this object at the same time. So, that is why we should have a lock here ? > Add MiniClusterClient to allow fast MiniCluster operations > -- > > Key: FLINK-8811 > URL: https://issues.apache.org/jira/browse/FLINK-8811 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.5.0, 1.6.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > Labels: flip-6 > Fix For: 1.5.0, 1.6.0 > > > We should offer a {{ClusterClient}} implementation for the {{MiniCluster}}. > That way we would be able to submit and wait for result without polling how > it would be the case by using the {{RestClusterClient}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8811) Add MiniClusterClient to allow fast MiniCluster operations
[ https://issues.apache.org/jira/browse/FLINK-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381915#comment-16381915 ] ASF GitHub Bot commented on FLINK-8811: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/5600 The error originates from the legacy code. Not sure what I could have changed there to be honest. Will see whether it passes locally or not. > Add MiniClusterClient to allow fast MiniCluster operations > -- > > Key: FLINK-8811 > URL: https://issues.apache.org/jira/browse/FLINK-8811 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.5.0, 1.6.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > Labels: flip-6 > Fix For: 1.5.0, 1.6.0 > > > We should offer a {{ClusterClient}} implementation for the {{MiniCluster}}. > That way we would be able to submit and wait for result without polling how > it would be the case by using the {{RestClusterClient}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8811) Add MiniClusterClient to allow fast MiniCluster operations
[ https://issues.apache.org/jira/browse/FLINK-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381771#comment-16381771 ] ASF GitHub Bot commented on FLINK-8811: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5600 +1 for merging this ASAP 👍 The error message seems a bit disconcerting, though: ``` Finished org.apache.flink.test.checkpointing.HAIncrementalRocksDbBackendEventTimeWindowCheckpointingITCase#testSlidingTimeWindow. org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:891) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:834) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Not enough free slots available to run the job. You can decrease the operator parallelism or increase the number of slots per TaskManager in the configuration. Task to schedule: < Attempt #3 (Source: Custom Source (1/1)) @ (unassigned) - [SCHEDULED] > with groupID < bc764cd8ddf7a0cff126f51c16239658 > in sharing group < 7c540813226d407fbf248f58c16b372c >. Resources available to scheduler: Number of instances=2, total number of slots=4, available slots=0 at org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleTask(Scheduler.java:281) at org.apache.flink.runtime.jobmanager.scheduler.Scheduler.allocateSlot(Scheduler.java:155) at org.apache.flink.runtime.executiongraph.Execution.lambda$allocateAndAssignSlotForExecution$1(Execution.java:489) at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) at org.apache.flink.runtime.executiongraph.Execution.allocateAndAssignSlotForExecution(Execution.java:487) at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.allocateResourcesForAll(ExecutionJobVertex.java:520) at org.apache.flink.runtime.executiongraph.ExecutionGraph.scheduleEager(ExecutionGraph.java:957) at org.apache.flink.runtime.executiongraph.ExecutionGraph.scheduleForExecution(ExecutionGraph.java:887) at org.apache.flink.runtime.executiongraph.ExecutionGraph.restart(ExecutionGraph.java:1282) at org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestartCallback.triggerFullRecovery(ExecutionGraphRestartCallback.java:59) at org.apache.flink.runtime.executiongraph.restart.FixedDelayRestartStrategy$1.run(FixedDelayRestartStrategy.java:68) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` > Add MiniClusterClient to allow fast MiniCluster operations > -- > > Key: FLINK-8811 > URL: https://issues.apache.org/jira/browse/FLINK-8811 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.5.0, 1.6.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > Labels: flip-6 > Fix For: 1.5.0, 1.6.0 > > > We should offer a {{ClusterClient}} implementation for the {{MiniCluster}}. > That way we would be able to su
[jira] [Commented] (FLINK-8811) Add MiniClusterClient to allow fast MiniCluster operations
[ https://issues.apache.org/jira/browse/FLINK-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380689#comment-16380689 ] ASF GitHub Bot commented on FLINK-8811: --- GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/5600 [FLINK-8811] [flip6] Add initial implementation of the MiniClusterClient ## What is the purpose of the change The MiniClusterClient directly talks to the MiniCluster avoiding polling latencies of th RestClusterClient. cc: @aljoscha ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink miniClusterClient Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5600.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5600 commit 308603035b54c46b0b3c6967138352694c563628 Author: Till Rohrmann Date: 2018-02-28T13:07:12Z [FLINK-8808] [flip6] Allow RestClusterClient to connect to local dispatcher The RestClusterClient resolves a dispatcher address without an explicit host to 'localhost'. That way we allow the RestClusterClient to talk to a Dispatcher which runs in a local ActorSystem. commit f52f37d26aae2ea57e3bf2f4b7e1e15a41a4529f Author: Till Rohrmann Date: 2018-02-28T16:36:39Z [FLINK-8811] [flip6] Add initial implementation of the MiniClusterClient The MiniClusterClient directly talks to the MiniCluster avoiding polling latencies of th RestClusterClient. commit caa6357f18eb1527e59c567cb3628df556b78f39 Author: Till Rohrmann Date: 2018-02-28T16:49:29Z [FLINK-8811] [flip6] Implement MiniClusterClient#getJobStatus commit 1088117c54f93aa314e249108237c2d2c04762d9 Author: Till Rohrmann Date: 2018-02-28T16:54:06Z [FLINK-8811] [flip6] Implement MiniClusterClient#cancel > Add MiniClusterClient to allow fast MiniCluster operations > -- > > Key: FLINK-8811 > URL: https://issues.apache.org/jira/browse/FLINK-8811 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.5.0, 1.6.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > Labels: flip-6 > Fix For: 1.5.0, 1.6.0 > > > We should offer a {{ClusterClient}} implementation for the {{MiniCluster}}. > That way we would be able to submit and wait for result without polling how > it would be the case by using the {{RestClusterClient}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)