[ https://issues.apache.org/jira/browse/FLINK-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269039#comment-16269039 ]
ASF GitHub Bot commented on FLINK-7918: --------------------------------------- GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/5095 [FLINK-7918] Run AbstractTestBase tests on Flip-6 MiniCluster ## What is the purpose of the change Extend `MiniClusterResource` to instantiate a Flip-6 `MiniCluster` if the system property `codebase` has been set to `flip6`. This allows to run all tests which are based on `AbstractTestBase` on the `Flip-6` code base. ## Brief change log - Use system property `codebase` to distinguish between Flip-6 and old code base - Instantiate respective `MiniCluster` in `MiniClusterResource` depending on the system property ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink flip6TestBase Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5095.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5095 ---- commit 1f6cb24a2b8e4b062fe12d756bfd5950ab35b0ce Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-11-14T22:50:52Z [FLINK-8078] Introduce LogicalSlot interface The LogicalSlot interface decouples the task deployment from the actual slot implementation which at the moment is Slot, SimpleSlot and SharedSlot. This is a helpful step to introduce a different slot implementation for Flip-6. commit aff11aaf02c7b5b667780031e01a1b08d85e036b Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-11-15T13:20:27Z [FLINK-8085] Thin out LogicalSlot interface Remove isCanceled, isReleased method and decouple logical slot from Execution by introducing a Payload interface which is set for a LogicalSlot. The Payload interface is implemented by the Execution and allows to fail an implementation and obtaining a termination future. Introduce proper Execution#releaseFuture which is completed once the Execution's assigned resource has been released. commit 040cb326086968b5fe5d66dcc82835ee9dd092dc Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-11-24T17:03:49Z [FLINK-8087] Decouple Slot from AllocatedSlot This commit introduces the SlotContext which is an abstraction for the SimpleSlot to obtain the relevant slot information to do the communication with the TaskManager without relying on the AllocatedSlot which is now only used by the SlotPool. commit 5d7e117392a51a485a5cb682c4a2c12028e73b97 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-11-24T17:06:10Z [FLINK-8088] Associate logical slots with the slot request id Before logical slots like the SimpleSlot and SharedSlot where associated to the actually allocated slot via the AllocationID. This, however, was sub-optimal because allocated slots can be re-used to fulfill also other slot requests (logical slots). Therefore, we should bind the logical slots to the right id with the right lifecycle which is the slot request id. commit 75f49d4bb3f38c7bfaee101607a459d558c00ba8 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-11-13T14:42:07Z [FLINK-8089] Also check for other pending slot requests in offerSlot Not only check for a slot request with the right allocation id but also check whether we can fulfill other pending slot requests with an unclaimed offered slot before adding it to the list of available slots. commit b53729e74ccc31c634c5ac7db5e37ca45a66db8d Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-11-24T17:08:38Z [FLINK-7956] [flip6] Add support for queued scheduling with slot sharing to SlotPool This commit adds support for queued scheduling with slot sharing to the SlotPool. The idea of slot sharing is that multiple tasks can run in the same slot. Moreover, queued scheduling means that a slot request must not be completed right away but at a later point in time. This allows to start new TaskExecutors in case that there are no more slots left. The main component responsible for the management of shared slots is the SlotSharingManager. The SlotSharingManager maintains internally a tree-like structure which stores the SlotContext future of the underlying AllocatedSlot. Whenever this future is completed potentially pending LogicalSlot instantiations are executed and sent to the slot requester. A shared slot is represented by a MultiTaskSlot which can harbour multiple TaskSlots. A TaskSlot can either be a MultiTaskSlot or a SingleTaskSlot. In order to represent co-location constraints, we first obtain a root MultiTaskSlot and then allocate a nested MultiTaskSlot in which the co-located tasks are allocated. The corresponding SlotRequestID is assigned to the CoLocationConstraint in order to make the TaskSlot retrievable for other tasks assigned to the same CoLocationConstraint. Port SchedulerSlotSharingTest, SchedulerIsolatedTasksTest and ScheduleWithCoLocationHintTest to run with SlotPool. Restructure SlotPool components. Add SlotSharingManagerTest, SlotPoolSlotSharingTest and SlotPoolCoLocationTest. commit b1d520efbe258070d92e9b66904e4e1d3d31fee6 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-11-24T22:29:53Z [hotfix] Speed up RecoveryITCase commit 5df0b85ed39edb35e22ae684306c6bc564ff02e4 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-23T14:21:14Z [FLINK-7903] [tests] Add flip6 build profile The flip6 build profile only runs the Flip-6 related test cases. Moreover, all Flip-6 related test cases are excluded when not running the flip6 build profile. This should reduce testing time when adding more and more Flip-6 test cases. commit df818af4a5b33cb9f9c8c1606aa715b08b926093 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-24T10:34:53Z Include flink-test-utils-junit in all submodules to make the Category marker interfaces Flip6 and OldAndFlip6 available commit ea8ac2f75ab176a9b6fd6491a289ef3f7433aff6 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-11-27T08:29:54Z [hotfix] [tests] Speed up queryable state IT tests by removing sleep commit 8344dde82f5c8a4307084f5a0558c4520a5cd880 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-23T14:24:58Z [FLINK-7904] Enable Flip6 build profile on Travis This adds a new Travis build matrix entry which runs the Flip-6 build profile commit 8bb13fd9cbea940e98552367128b9dde70ab80d0 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-24T13:38:02Z Add core+tests TEST option commit 53c15ce1d9deb2a8060764da75365df052ad7ed9 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-24T13:51:27Z [hotfix] Remove unnecessary surefire plugin version in flink-connector-elasticsearch5 commit 16b5caab9422ec1d60192bd1adba2c725ad0bab0 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-25T10:03:31Z Reuse caches and split core+test into core and tests Travis build commit 76fe0d635c6a0a41c47317bd1dd9c7e887886ca2 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-24T09:32:05Z [FLINK-7909] Unify Flink test bases Introduce a MiniClusterResource which is used by the AbstractTestBase to start and shut down a FlinkMiniCluster. Additionally, this resource registers the proper Stream- and ExecutionEnvironment which is now the only way for tests to start jobs. This change will thus allow to centrally control which FlinkCluster will be started for all test bases. commit 2a5015d985e8f40695b4a6676772f32c0a7602f3 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-24T16:53:40Z [FLINK-7910] [tests] Generalize Test(Stream)Environment to use JobExecutor This commit introduces the JobExecutor interface which abstracts the actual mini cluster from the Test(Stream)Environment. By letting the Flip-6 MiniCluster as well as the FlinkMiniCluster implement this interface, we can run all test base jobs either on the Flip-6 mini cluster or on the current mini cluster. commit 3395d3041caedd754e2414c8d03b4f3d47f2ba59 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-25T11:29:38Z Fix travis.yml commit a2f9734463df2eafd05f48e89563611adbd549f3 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-24T14:20:15Z [FLINK-7909] Replace StreamingMultipleProgramsTestBase by AbstractTestBase The AbstractTestBase fully subsumes the functionality of the StreamingMultipleProgramsTestBase since it now is the most general test base for streaming and batch jobs. As a consequence, we can safely remove the StreamingMultipleProgramsTestBase and let all corresponding tests extend from AbstractTestBase. commit 87a188dab5af31d35251fd644bda63040cc559f0 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-25T09:43:06Z Fix failing HadoopCompatibility test cases commit 9fd8761393c9fb17094eb95f9603a2cc58855e66 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-25T10:08:29Z Fix flink-connector-filesystem commit 7d9c1f285866d54076f877419ae1c7ab9ecd1ca5 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-25T12:55:06Z Fix failing test cases commit 0be28d9cb940488dc204bb058bda2c6c04d619bf Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-25T13:13:39Z Fix commit d8c8c8e9cf9634f66da33100d3ef473886ef4f81 Author: Till Rohrmann <trohrm...@apache.org> Date: 2017-10-23T15:02:58Z [FLINK-7918] Run AbstractTestBase tests on Flip-6 MiniCluster ---- > Run AbstractTestBase tests on Flip-6 MiniCluster > ------------------------------------------------ > > Key: FLINK-7918 > URL: https://issues.apache.org/jira/browse/FLINK-7918 > Project: Flink > Issue Type: New Feature > Components: Tests > Affects Versions: 1.4.0 > Reporter: Till Rohrmann > Assignee: Till Rohrmann > Labels: flip-6 > > In order to extend our test coverage for Flip-6, we should enable all > {{AbstractTestBase}} tests to be executed via Flip-6's {{MiniCluster}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029)