GitHub user ankurcha opened a pull request:
https://github.com/apache/flink/pull/948
[FLINK-1984] Integrate Flink with Apache Mesos
This pull requests adds a mesos scheduler and an executor (inspired from
the work done in, now abandoned, PR #251). The highlights are as follows:
* The mesos scheduler is starts a jobManager in a parallel thread based on
the configuration provided using `--configDir` argument.
* The mesos scheduler is not HA and should be used with marathon or similar
service to ensure that there is always one instance running. This may be
addressed in future patches.
* The mesos scheduler uses the some new properties which can be set using
the `conf/flink-conf.yaml`. The configuration directory can be specified using
the `--confDir` command line argument and a list of configuration values is
present in
`flink-mesos/src/main/scala/org/apache/flink/mesos/scheduler/package.scala`.
Example usage:
```bash
mvn clean package -DskipTests -Pinclude-mesos
java -Dlog4j.configuration=file:/vagrant/log4j.properties -cp *.jar
org.apache.flink.mesos.scheduler.FlinkScheduler --conf-dir /vagrant
```
```yaml
# flink-conf.yaml
flink.mesos.master: zk://127.0.0.1:2181/mesos
flink.uberjar.location: file:///vagrant/flink-dist-0.10-SNAPSHOT.jar
flink.mesos.taskmanagers.mem: 512
flink.mesos.taskmanagers.cpu: 0.5
taskmanager.logging.level: INFO
streamingMode: streaming
jobmanager.web.port: 8081
webclient.port: 8080
```
```
# log4j.properties
log4j.rootLogger=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%-5p %d{ISO8601} [%t]
%c{1}: %m%n
# suppress the warning that hadoop native libraries are not loaded
(irrelevant for the client)
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=OFF
log4j.logger.org.apache.flink.mesos=DEBUG
# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, console
```
An easy way to test this is using https://github.com/mesosphere/playa-mesos
to start a mesos cluster in virtualbox (using vagrant) and copying the uberjar
to /vagrant and running the above command. The job manager web UI url is set as
the mesos framework url.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ankurcha/flink flink-mesos
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/948.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #948
----
commit 8a9e3791a2c3946f1adefe64845f16f89e092c77
Author: Ankur Chauhan <[email protected]>
Date: 2015-07-20T08:45:41Z
wip
commit a937ff65d72a938814162de1e9d97640c566b3f1
Author: Ankur Chauhan <[email protected]>
Date: 2015-07-25T19:18:42Z
WIP - flink mesos integration initial commit
commit 3f0237adb7b1f45a1fd80e025aa5b511f92a209c
Author: Ankur Chauhan <[email protected]>
Date: 2015-07-25T19:33:57Z
Add todo for config changes/issues
commit d9c29e2a3174cb90626c9deccc1604e0e010d965
Author: Ankur Chauhan <[email protected]>
Date: 2015-07-27T00:03:12Z
Fix some stuff
commit 9910f50c11c81631ff8bb351cdf394cda753ff98
Author: Ankur Chauhan <[email protected]>
Date: 2015-07-28T01:41:20Z
Add flink-mesos to dist, update scheduler and executor code
commit 4f2d8fed5b2bf8f49eb4da7cd148e83016206330
Author: Ankur Chauhan <[email protected]>
Date: 2015-07-28T03:14:16Z
Clean up code + codestyle changes
commit b1613a965a59f89aa2e9c96dcec52fc40847423a
Author: Ankur Chauhan <[email protected]>
Date: 2015-07-28T07:12:55Z
Clean up executor, adjust resource defaults
commit 5d39b1524b9ec565c5be28abcd4f80fab928042b
Author: Ankur Chauhan <[email protected]>
Date: 2015-07-28T20:56:24Z
Remove status pinging and fix logging level in TaskManagerExecutor
commit 3331596c957ad774ffca31bb2c959b22422560d2
Author: Ankur Chauhan <[email protected]>
Date: 2015-07-28T22:33:54Z
Code cleanup and refactoring of the conf classes
commit 33f3597f1c2aa0c7582025d9c242d4aafd9cfa63
Author: Ankur Chauhan <[email protected]>
Date: 2015-07-28T22:36:04Z
Remove unused dependencies and plugins
commit 36846a95a64b9700d37954a2d0176b222b6adbe0
Author: Ankur Chauhan <[email protected]>
Date: 2015-07-28T22:40:50Z
Use tab based indents in pom files
commit 52904c8593377c349bccf599b863f2a40df0f562
Author: Ankur Chauhan <[email protected]>
Date: 2015-07-28T23:06:30Z
Add comments for each of the properties
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
