[
https://issues.apache.org/jira/browse/HADOOP-16063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota reassigned HADOOP-16063:
-----------------------------------
Assignee: (was: Gabor Bota)
> Docker based pseudo-cluster definitions and test scripts for Hdfs/Yarn
> ----------------------------------------------------------------------
>
> Key: HADOOP-16063
> URL: https://issues.apache.org/jira/browse/HADOOP-16063
> Project: Hadoop Common
> Issue Type: New Feature
> Reporter: Marton Elek
> Priority: Major
>
> During the recent releases of Apache Hadoop Ozone we had multiple experiments
> using docker/docker-compose to support the development of ozone.
> As of now the hadoop-ozone distribution contains two directories in
> additional the regular hadoop directories (bin, share/lib, etc
> h3. compose
> The ./compose directory of the distribution contains different type of
> pseudo-cluster definitions. To start an ozone cluster is as easy as "cd
> compose/ozone && docker-compose up-d"
> The clusters also could be scaled up and down (docker-compose scale
> datanode=3)
> There are multiple cluster definitions for different use cases (for example
> ozone+s3 or hdfs+ozone).
> The docker-compose files are based on apache/hadoop-runner image which is an
> "empty" image. It doesnt' contain any hadoop distribution. Instead the
> current hadoop is used (the ../.. is mapped as a volume at /opt/hadoop)
> With this approach it's very easy to 1) start a cluster from the distribution
> 2) test any patch from the dev tree, as after any build a new cluster can be
> started easily (with multiple nodes and datanodes)
> h3. smoketest
> We also started to use a simple robotframework based test suite. (see
> ./smoketest directory). It's a high level test definition very similar to the
> smoketests which are executed manually by the contributors during a release
> vote.
> But it's a formal definition to start cluster from different docker-compose
> definitions and execute simple shell scripts (and compare the output).
>
> I believe that both approaches helped a lot during the development of ozone
> and I propose to do the same improvements on the main hadoop distribution.
> I propose to provide docker-compose based example cluster definitions for
> yarn/hdfs and for different use cases (simple hdfs, router based federation,
> etc.)
> It can help to understand the different configuration and try out new
> features with predefined config set.
> Long term we can also add robottests to help the release votes (basic
> wordcount/mr tests could be scripted)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
