Evans Ye created BIGTOP-2397:
--------------------------------
Summary: Add image pre-build new feature
Key: BIGTOP-2397
URL: https://issues.apache.org/jira/browse/BIGTOP-2397
Project: Bigtop
Issue Type: Sub-task
Components: docker, provisioner
Affects Versions: backlog
Reporter: Evans Ye
Assignee: Evans Ye
Currently the provisioner takes roughly 3 to 5 mins to get a cluster up and
running. This is not quite efficient for CI and developers to get a cluster for
use. This jira trys to add an image pre-build feature into docker provisioner
so that users can build the image once and run it multiple times. With this we
can expect a substantial performance boost for provisioning.
BIGTOP-2296 showcased a solution which is to pre-build a hard-coded stack,
i.e., hadoop + yarn. However, in this jira we'd like to make it more general,
for example, user can define a stack named "in-memory stack", or "foobar
company big data stack", which preload a set of components such as hadoop,
yarn, spark into a image for further provisioning.
The following is the more detailed steps for a cluster to be deployed with
pre-build feature involved:
# user operation stage: update configuration file and run the provisioner
command
{code}
vim config.yml
# specify image name, for example *foobar_company:evans-hadoop-stack*, and the
components
./docker-hadoop.sh --burn --create 3
# with --burn the image
{code}
# pre-build stage1: install general system packages such as java
# pre-build stage2: install stack packages such as hadoop, hbase, spark, etc
# provision stage1: upload configuration files
# provision stage2: run puppet apply to simply do deploy config files and start
up services
When user want to re-provision a clean cluster, the prebuild image will be used:
{code}
./docker-hadoop.sh --create 3
{code}
There's actually no code change at the provisioner side, but with packages are
already installed, the provision should be very fast.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
