GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/5095
[FLINK-7918] Run AbstractTestBase tests on Flip-6 MiniCluster
## What is the purpose of the change
Extend `MiniClusterResource` to instantiate a Flip-6 `MiniCluster` if the
system property `codebase` has been set to `flip6`. This allows to run all
tests which are based on `AbstractTestBase` on the `Flip-6` code base.
## Brief change log
- Use system property `codebase` to distinguish between Flip-6 and old code
base
- Instantiate respective `MiniCluster` in `MiniClusterResource` depending
on the system property
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink flip6TestBase
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5095.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5095
----
commit 1f6cb24a2b8e4b062fe12d756bfd5950ab35b0ce
Author: Till Rohrmann <[email protected]>
Date: 2017-11-14T22:50:52Z
[FLINK-8078] Introduce LogicalSlot interface
The LogicalSlot interface decouples the task deployment from the actual
slot implementation which at the moment is Slot, SimpleSlot and SharedSlot.
This is a helpful step to introduce a different slot implementation for
Flip-6.
commit aff11aaf02c7b5b667780031e01a1b08d85e036b
Author: Till Rohrmann <[email protected]>
Date: 2017-11-15T13:20:27Z
[FLINK-8085] Thin out LogicalSlot interface
Remove isCanceled, isReleased method and decouple logical slot from
Execution by
introducing a Payload interface which is set for a LogicalSlot. The Payload
interface
is implemented by the Execution and allows to fail an implementation and
obtaining
a termination future.
Introduce proper Execution#releaseFuture which is completed once the
Execution's
assigned resource has been released.
commit 040cb326086968b5fe5d66dcc82835ee9dd092dc
Author: Till Rohrmann <[email protected]>
Date: 2017-11-24T17:03:49Z
[FLINK-8087] Decouple Slot from AllocatedSlot
This commit introduces the SlotContext which is an abstraction for the
SimpleSlot
to obtain the relevant slot information to do the communication with the
TaskManager without relying on the AllocatedSlot which is now only used by
the
SlotPool.
commit 5d7e117392a51a485a5cb682c4a2c12028e73b97
Author: Till Rohrmann <[email protected]>
Date: 2017-11-24T17:06:10Z
[FLINK-8088] Associate logical slots with the slot request id
Before logical slots like the SimpleSlot and SharedSlot where associated to
the
actually allocated slot via the AllocationID. This, however, was
sub-optimal because
allocated slots can be re-used to fulfill also other slot requests (logical
slots).
Therefore, we should bind the logical slots to the right id with the right
lifecycle
which is the slot request id.
commit 75f49d4bb3f38c7bfaee101607a459d558c00ba8
Author: Till Rohrmann <[email protected]>
Date: 2017-11-13T14:42:07Z
[FLINK-8089] Also check for other pending slot requests in offerSlot
Not only check for a slot request with the right allocation id but also
check
whether we can fulfill other pending slot requests with an unclaimed offered
slot before adding it to the list of available slots.
commit b53729e74ccc31c634c5ac7db5e37ca45a66db8d
Author: Till Rohrmann <[email protected]>
Date: 2017-11-24T17:08:38Z
[FLINK-7956] [flip6] Add support for queued scheduling with slot sharing to
SlotPool
This commit adds support for queued scheduling with slot sharing to the
SlotPool. The idea of slot sharing is that multiple tasks can run in the
same slot. Moreover, queued scheduling means that a slot request must not
be completed right away but at a later point in time. This allows to
start new TaskExecutors in case that there are no more slots left.
The main component responsible for the management of shared slots is the
SlotSharingManager. The SlotSharingManager maintains internally a tree-like
structure which stores the SlotContext future of the underlying
AllocatedSlot. Whenever this future is completed potentially pending
LogicalSlot instantiations are executed and sent to the slot requester.
A shared slot is represented by a MultiTaskSlot which can harbour multiple
TaskSlots. A TaskSlot can either be a MultiTaskSlot or a SingleTaskSlot.
In order to represent co-location constraints, we first obtain a root
MultiTaskSlot and then allocate a nested MultiTaskSlot in which the
co-located tasks are allocated. The corresponding SlotRequestID is assigned
to the CoLocationConstraint in order to make the TaskSlot retrievable for
other tasks assigned to the same CoLocationConstraint.
Port SchedulerSlotSharingTest, SchedulerIsolatedTasksTest and
ScheduleWithCoLocationHintTest to run with SlotPool.
Restructure SlotPool components.
Add SlotSharingManagerTest, SlotPoolSlotSharingTest and
SlotPoolCoLocationTest.
commit b1d520efbe258070d92e9b66904e4e1d3d31fee6
Author: Till Rohrmann <[email protected]>
Date: 2017-11-24T22:29:53Z
[hotfix] Speed up RecoveryITCase
commit 5df0b85ed39edb35e22ae684306c6bc564ff02e4
Author: Till Rohrmann <[email protected]>
Date: 2017-10-23T14:21:14Z
[FLINK-7903] [tests] Add flip6 build profile
The flip6 build profile only runs the Flip-6 related test cases. Moreover,
all Flip-6 related test cases are excluded when not running the flip6 build
profile. This should reduce testing time when adding more and more Flip-6
test cases.
commit df818af4a5b33cb9f9c8c1606aa715b08b926093
Author: Till Rohrmann <[email protected]>
Date: 2017-10-24T10:34:53Z
Include flink-test-utils-junit in all submodules to make the Category
marker interfaces Flip6 and OldAndFlip6 available
commit ea8ac2f75ab176a9b6fd6491a289ef3f7433aff6
Author: Till Rohrmann <[email protected]>
Date: 2017-11-27T08:29:54Z
[hotfix] [tests] Speed up queryable state IT tests by removing sleep
commit 8344dde82f5c8a4307084f5a0558c4520a5cd880
Author: Till Rohrmann <[email protected]>
Date: 2017-10-23T14:24:58Z
[FLINK-7904] Enable Flip6 build profile on Travis
This adds a new Travis build matrix entry which runs the Flip-6 build
profile
commit 8bb13fd9cbea940e98552367128b9dde70ab80d0
Author: Till Rohrmann <[email protected]>
Date: 2017-10-24T13:38:02Z
Add core+tests TEST option
commit 53c15ce1d9deb2a8060764da75365df052ad7ed9
Author: Till Rohrmann <[email protected]>
Date: 2017-10-24T13:51:27Z
[hotfix] Remove unnecessary surefire plugin version in
flink-connector-elasticsearch5
commit 16b5caab9422ec1d60192bd1adba2c725ad0bab0
Author: Till Rohrmann <[email protected]>
Date: 2017-10-25T10:03:31Z
Reuse caches and split core+test into core and tests Travis build
commit 76fe0d635c6a0a41c47317bd1dd9c7e887886ca2
Author: Till Rohrmann <[email protected]>
Date: 2017-10-24T09:32:05Z
[FLINK-7909] Unify Flink test bases
Introduce a MiniClusterResource which is used by the AbstractTestBase to
start
and shut down a FlinkMiniCluster. Additionally, this resource registers the
proper
Stream- and ExecutionEnvironment which is now the only way for tests to
start
jobs. This change will thus allow to centrally control which FlinkCluster
will
be started for all test bases.
commit 2a5015d985e8f40695b4a6676772f32c0a7602f3
Author: Till Rohrmann <[email protected]>
Date: 2017-10-24T16:53:40Z
[FLINK-7910] [tests] Generalize Test(Stream)Environment to use JobExecutor
This commit introduces the JobExecutor interface which abstracts the actual
mini cluster
from the Test(Stream)Environment. By letting the Flip-6 MiniCluster as well
as the
FlinkMiniCluster implement this interface, we can run all test base jobs
either on the
Flip-6 mini cluster or on the current mini cluster.
commit 3395d3041caedd754e2414c8d03b4f3d47f2ba59
Author: Till Rohrmann <[email protected]>
Date: 2017-10-25T11:29:38Z
Fix travis.yml
commit a2f9734463df2eafd05f48e89563611adbd549f3
Author: Till Rohrmann <[email protected]>
Date: 2017-10-24T14:20:15Z
[FLINK-7909] Replace StreamingMultipleProgramsTestBase by AbstractTestBase
The AbstractTestBase fully subsumes the functionality of the
StreamingMultipleProgramsTestBase since it now is the most general test base
for streaming and batch jobs. As a consequence, we can safely remove the
StreamingMultipleProgramsTestBase and let all corresponding tests extend
from
AbstractTestBase.
commit 87a188dab5af31d35251fd644bda63040cc559f0
Author: Till Rohrmann <[email protected]>
Date: 2017-10-25T09:43:06Z
Fix failing HadoopCompatibility test cases
commit 9fd8761393c9fb17094eb95f9603a2cc58855e66
Author: Till Rohrmann <[email protected]>
Date: 2017-10-25T10:08:29Z
Fix flink-connector-filesystem
commit 7d9c1f285866d54076f877419ae1c7ab9ecd1ca5
Author: Till Rohrmann <[email protected]>
Date: 2017-10-25T12:55:06Z
Fix failing test cases
commit 0be28d9cb940488dc204bb058bda2c6c04d619bf
Author: Till Rohrmann <[email protected]>
Date: 2017-10-25T13:13:39Z
Fix
commit d8c8c8e9cf9634f66da33100d3ef473886ef4f81
Author: Till Rohrmann <[email protected]>
Date: 2017-10-23T15:02:58Z
[FLINK-7918] Run AbstractTestBase tests on Flip-6 MiniCluster
----
---
