METRON-1351 Create Installable Packages for Ubuntu Trusty (nickwallen) closes apache/metron#868
Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/fc8723e4 Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/fc8723e4 Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/fc8723e4 Branch: refs/heads/feature/METRON-1211-extensions-parsers-gradual Commit: fc8723e461d655e315d0b51acd1a31f82b4efd1f Parents: 3612a89 Author: nickwallen <[email protected]> Authored: Wed Dec 27 13:25:53 2017 -0500 Committer: nickallen <[email protected]> Committed: Wed Dec 27 13:25:53 2017 -0500 ---------------------------------------------------------------------- .../packaging/docker/deb-docker/.gitignore | 2 + .../packaging/docker/deb-docker/Dockerfile | 33 +++ .../packaging/docker/deb-docker/README.md | 43 ++++ .../packaging/docker/deb-docker/build.sh | 72 ++++++ .../docker/deb-docker/debian/changelog | 22 ++ .../packaging/docker/deb-docker/debian/control | 28 ++ .../docker/deb-docker/debian/copyright | 35 +++ .../packaging/docker/deb-docker/pom.xml | 253 +++++++++++++++++++ .../docker/deb-docker/prepackage/metron-config | 32 +++ .../packaging/docker/rpm-docker/.gitignore | 2 +- metron-deployment/pom.xml | 6 + 11 files changed, 527 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron/blob/fc8723e4/metron-deployment/packaging/docker/deb-docker/.gitignore ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/docker/deb-docker/.gitignore b/metron-deployment/packaging/docker/deb-docker/.gitignore new file mode 100644 index 0000000..8162550 --- /dev/null +++ b/metron-deployment/packaging/docker/deb-docker/.gitignore @@ -0,0 +1,2 @@ +.* +target http://git-wip-us.apache.org/repos/asf/metron/blob/fc8723e4/metron-deployment/packaging/docker/deb-docker/Dockerfile ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/docker/deb-docker/Dockerfile b/metron-deployment/packaging/docker/deb-docker/Dockerfile new file mode 100644 index 0000000..44203c6 --- /dev/null +++ b/metron-deployment/packaging/docker/deb-docker/Dockerfile @@ -0,0 +1,33 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +FROM ubuntu:trusty + +# install prerequisites +RUN apt-get update && apt-get install -y \ + build-essential \ + openjdk-7-jdk \ + wget \ + curl \ + dpkg-dev \ + gettext + +# install nodejs so that the node dependencies can be packaged into the DEBs +RUN curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash - +RUN apt-get install -y nodejs + +WORKDIR /root http://git-wip-us.apache.org/repos/asf/metron/blob/fc8723e4/metron-deployment/packaging/docker/deb-docker/README.md ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/docker/deb-docker/README.md b/metron-deployment/packaging/docker/deb-docker/README.md new file mode 100644 index 0000000..8ee8671 --- /dev/null +++ b/metron-deployment/packaging/docker/deb-docker/README.md @@ -0,0 +1,43 @@ + +This project builds DEB packages for installing Apache Metron on Ubuntu. + +**WARNING**: The DEB packages are a recent addition to Metron. These packages have not undergone the same level of testing as the RPM packages. Improvements and more rigerous testing of these packages is underway and will improve in future releases. Until then, use these at your own risk. + +### Prerequisites + +Building these packages requires Docker. Please ensure that Docker is installed and that the daemon is running. + +### How do I create packages for Ubuntu? + +Running the following from Metron's top-level source directory will build Metron along with the DEB packages. Using the `package` or `install` target are both sufficient. + + ``` + mvn clean package -DskipTests -Pbuild-debs + ``` + +If Metron has already been built, just the DEB packages can be built by doing the following. + + ``` + cd metron-deployment + mvn clean package -Pbuild-debs + ``` + +### How does this really work? + +Using the `build-debs` profile as shown above effectively automates the following steps. + +1. Copy the tarball for each Metron sub-project to the `target` working directory. + +1. Build a Docker image of a Ubuntu Trusty host called `docker-deb` that contains all of the tools needed to build the packages. + + ``` + docker build -t deb-docker . + ``` + +1. Execute the `build.sh` script within a Docker container. The argument passed to the build script is the current version of Metron. + + ``` + docker run -v `pwd`:/root deb-docker:latest /bin/bash -c ./build.sh 0.4.2 + ``` + +1. This results in the DEBs being generated within the `target` directory. http://git-wip-us.apache.org/repos/asf/metron/blob/fc8723e4/metron-deployment/packaging/docker/deb-docker/build.sh ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/docker/deb-docker/build.sh b/metron-deployment/packaging/docker/deb-docker/build.sh new file mode 100755 index 0000000..0a79841 --- /dev/null +++ b/metron-deployment/packaging/docker/deb-docker/build.sh @@ -0,0 +1,72 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +export FULL_VERSION=$1 +export VERSION=$(echo ${FULL_VERSION} | tr -d '"'"'[:alpha:]'"'"') +export DISTRIBUTION="trusty" + +echo "FULL_VERSION: ${FULL_VERSION}" +echo "VERSION: ${VERSION}" + +INSTALL_PREFIX="/usr/metron" +METRON_HOME="${INSTALL_PREFIX}/${FULL_VERSION}" +HOMEDIR="/root" +WORKDIR="${HOMEDIR}/target" +CONFIGDIR="${HOMEDIR}/debian/" +DATE=`date '+%Y%m%d%H%M'` +PREPACKAGEDIR="${HOMEDIR}/prepackage" + +# working directory +mkdir -p "${WORKDIR}" +cd "${WORKDIR}" + +# for each metron tarball... +for TARBALL in metron*.tar.gz; do + + export PACKAGE=`echo ${TARBALL} | awk -F"-${FULL_VERSION}-" '{print $1}'` + export PACKAGE_WORKDIR="${WORKDIR}/${PACKAGE}_${FULL_VERSION}" + echo "Building package; name=${PACKAGE}, tarball=${TARBALL}" + + # extract the package contents + mkdir -p ${PACKAGE_WORKDIR}/${METRON_HOME} + tar xf ${TARBALL} -C ${PACKAGE_WORKDIR}/${METRON_HOME} + rm -f ${TARBALL} + + # create the config directory + PACKAGE_DEBIAN_DIR="${PACKAGE_WORKDIR}/DEBIAN" + mkdir ${PACKAGE_DEBIAN_DIR} + + # create the configuration files + envsubst < ${CONFIGDIR}/control > ${PACKAGE_DEBIAN_DIR}/control + envsubst < ${CONFIGDIR}/changelog > ${PACKAGE_DEBIAN_DIR}/changelog + envsubst < ${CONFIGDIR}/copyright > ${PACKAGE_DEBIAN_DIR}/copyright + + # strip comments from the config files + sed -i 's/#.*$//g' ${PACKAGE_DEBIAN_DIR}/control + sed -i 's/#.*$//g' ${PACKAGE_DEBIAN_DIR}/changelog + sed -i 's/#.*$//g' ${PACKAGE_DEBIAN_DIR}/copyright + + # execute the prepackage script, if one exists + if [ -f "${PREPACKAGEDIR}/${PACKAGE}" ]; then + source ${PREPACKAGEDIR}/${PACKAGE} + fi + + # create the package + dpkg-deb --verbose --build ${PACKAGE_WORKDIR} + rm -rf ${PACKAGE_WORKDIR} +done http://git-wip-us.apache.org/repos/asf/metron/blob/fc8723e4/metron-deployment/packaging/docker/deb-docker/debian/changelog ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/docker/deb-docker/debian/changelog b/metron-deployment/packaging/docker/deb-docker/debian/changelog new file mode 100644 index 0000000..8b271bf --- /dev/null +++ b/metron-deployment/packaging/docker/deb-docker/debian/changelog @@ -0,0 +1,22 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +$PACKAGE ($FULL_VERSION) $DISTRIBUTION; urgency=medium + + * Initial release. (Closes: METRON-1351) + + -- Apache Metron <[email protected]> Wed, 13 Dec 2017 21:19:45 +0000 http://git-wip-us.apache.org/repos/asf/metron/blob/fc8723e4/metron-deployment/packaging/docker/deb-docker/debian/control ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/docker/deb-docker/debian/control b/metron-deployment/packaging/docker/deb-docker/debian/control new file mode 100644 index 0000000..febd3a9 --- /dev/null +++ b/metron-deployment/packaging/docker/deb-docker/debian/control @@ -0,0 +1,28 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +Source: metron +Section: misc +Priority: extra +Maintainer: Apache Metron <[email protected]> +Homepage: https://metron.apache.org/ +Package: $PACKAGE +Architecture: all +Version: $FULL_VERSION +Depends: +Description: Apache Metron + Apache Metron provides a scalable advanced security analytics framework. http://git-wip-us.apache.org/repos/asf/metron/blob/fc8723e4/metron-deployment/packaging/docker/deb-docker/debian/copyright ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/docker/deb-docker/debian/copyright b/metron-deployment/packaging/docker/deb-docker/debian/copyright new file mode 100644 index 0000000..d02dfb0 --- /dev/null +++ b/metron-deployment/packaging/docker/deb-docker/debian/copyright @@ -0,0 +1,35 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ +Upstream-Name: $PACKAGE +Upstream-Contact: Apache Metron <[email protected]> + +Files: * +License: ASL-2 +Copyright: +Apache Metron +Copyright 2015-2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +This product includes software developed by Cisco Systems (http://www.cisco.com) +Copyright (c) 2014 Cisco Systems. + +This product includes software developed by Chef Software (https://www.chef.io) +Copyright (c) 2012-2015, Chef Software, Inc. http://git-wip-us.apache.org/repos/asf/metron/blob/fc8723e4/metron-deployment/packaging/docker/deb-docker/pom.xml ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/docker/deb-docker/pom.xml b/metron-deployment/packaging/docker/deb-docker/pom.xml new file mode 100644 index 0000000..6cf8ca7 --- /dev/null +++ b/metron-deployment/packaging/docker/deb-docker/pom.xml @@ -0,0 +1,253 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <artifactId>metron-debs</artifactId> + <packaging>pom</packaging> + <name>metron-debs</name> + <version>0.4.2</version> + <parent> + <groupId>org.apache.metron</groupId> + <artifactId>metron-deployment</artifactId> + <version>0.4.2</version> + <relativePath>../../..</relativePath> + </parent> + <description>DEB Package Builder for Apache Metron</description> + <properties> + <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> + <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> + <docker.tag>deb-docker</docker.tag> + <metron_dir>${project.parent.relativePath}/..</metron_dir> + </properties> + <build> + <plugins> + <plugin> + <artifactId>maven-clean-plugin</artifactId> + <version>3.0.0</version> + <configuration> + <filesets> + <fileset> + <directory>target</directory> + <followSymlinks>false</followSymlinks> + <includes> + <include>**/*</include> + </includes> + </fileset> + </filesets> + </configuration> + </plugin> + <plugin> + <artifactId>maven-resources-plugin</artifactId> + <version>3.0.1</version> + <executions> + <execution> + <id>copy-deb-sources</id> + <phase>prepare-package</phase> + <goals> + <goal>copy-resources</goal> + </goals> + <configuration> + <outputDirectory>${project.basedir}/target</outputDirectory> + <resources> + <resource> + <directory>${metron_dir}/metron-platform/metron-common/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + <resource> + <directory>${metron_dir}/metron-platform/metron-data-management/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + <resource> + <directory>${metron_dir}/metron-platform/metron-elasticsearch/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + <resource> + <directory>${metron_dir}/metron-platform/metron-enrichment/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + <resource> + <directory>${metron_dir}/metron-platform/metron-indexing/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + <resource> + <directory>${metron_dir}/metron-platform/metron-management/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + <resource> + <directory>${metron_dir}/metron-platform/metron-parsers/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + <resource> + <directory>${metron_dir}/metron-platform/metron-pcap-backend/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + <resource> + <directory>${metron_dir}/metron-platform/metron-solr/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + <resource> + <directory>${metron_dir}/metron-analytics/metron-profiler/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + <resource> + <directory>${metron_dir}/metron-interface/metron-rest/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + <resource> + <directory>${metron_dir}/metron-interface/metron-config/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + <resource> + <directory>${metron_dir}/metron-analytics/metron-maas-service/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + <resource> + <directory>${metron_dir}/metron-interface/metron-alerts/target/</directory> + <includes> + <include>*.tar.gz</include> + </includes> + </resource> + </resources> + </configuration> + </execution> + </executions> + </plugin> + </plugins> + </build> + <profiles> + <profile> + <id>InDocker</id> + <activation> + <file> + <!-- If this file exists, then we are running in docker already --> + <exists>/.dockerenv</exists> + </file> + </activation> + <build> + <plugins> + <plugin> + <groupId>org.codehaus.mojo</groupId> + <artifactId>exec-maven-plugin</artifactId> + <version>1.5.0</version> + <executions> + <execution> + <id>deb-build</id> + <phase>package</phase> + <goals> + <goal>exec</goal> + </goals> + <configuration> + <executable>/bin/bash</executable> + <arguments> + <argument>-x</argument> + <argument>-c</argument> + <argument>./build.sh ${project.version}</argument> + </arguments> + </configuration> + </execution> + </executions> + </plugin> + </plugins> + </build> + </profile> + <profile> + <id>NeedsDocker</id> + <activation> + <file> + <!-- if this file doesn't exist, then we need to run docker --> + <missing>/.dockerenv</missing> + </file> + </activation> + <build> + <plugins> + <plugin> + <groupId>org.codehaus.mojo</groupId> + <artifactId>exec-maven-plugin</artifactId> + <version>1.5.0</version> + <executions> + <execution> + <id>docker-build</id> + <phase>package</phase> + <goals> + <goal>exec</goal> + </goals> + <configuration> + <executable>docker</executable> + <arguments> + <argument>build</argument> + <argument>-f</argument> + <argument>Dockerfile</argument> + <argument>-t</argument> + <argument>${docker.tag}</argument> + <argument>.</argument> + </arguments> + </configuration> + </execution> + <execution> + <id>deb-build</id> + <phase>package</phase> + <goals> + <goal>exec</goal> + </goals> + <configuration> + <executable>docker</executable> + <arguments> + <argument>run</argument> + <argument>-v</argument> + <argument>${project.basedir}:/root</argument> + <argument>${docker.tag}:latest</argument> + <argument>/bin/bash</argument> + <argument>-c</argument> + <argument>./build.sh ${project.version}</argument> + </arguments> + </configuration> + </execution> + </executions> + </plugin> + </plugins> + </build> + </profile> + </profiles> +</project> http://git-wip-us.apache.org/repos/asf/metron/blob/fc8723e4/metron-deployment/packaging/docker/deb-docker/prepackage/metron-config ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/docker/deb-docker/prepackage/metron-config b/metron-deployment/packaging/docker/deb-docker/prepackage/metron-config new file mode 100644 index 0000000..a5ac74f --- /dev/null +++ b/metron-deployment/packaging/docker/deb-docker/prepackage/metron-config @@ -0,0 +1,32 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# +# this script will be executed before packaging 'metron-config' +# +# this is called by the main `build.sh` script right after it has +# prepared all of the necessary files and configuration, but before +# packaging. this lets us perform any package-specific actions prior to +# final packaging. +# +# the working directory containing all of the files to +# be packaged will be passed as the only argument to this script. +# + +# package the 'production' node dependencies +npm install --prefix="${PACKAGE_WORKDIR}/${METRON_HOME}/web/expressjs" --only=production http://git-wip-us.apache.org/repos/asf/metron/blob/fc8723e4/metron-deployment/packaging/docker/rpm-docker/.gitignore ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/docker/rpm-docker/.gitignore b/metron-deployment/packaging/docker/rpm-docker/.gitignore index 75fbc06..d2e5db1 100644 --- a/metron-deployment/packaging/docker/rpm-docker/.gitignore +++ b/metron-deployment/packaging/docker/rpm-docker/.gitignore @@ -3,4 +3,4 @@ BUILDROOT/ RPMS/ SOURCES/ SRPMS/ -.npm +.* http://git-wip-us.apache.org/repos/asf/metron/blob/fc8723e4/metron-deployment/pom.xml ---------------------------------------------------------------------- diff --git a/metron-deployment/pom.xml b/metron-deployment/pom.xml index 20eb848..bc69142 100644 --- a/metron-deployment/pom.xml +++ b/metron-deployment/pom.xml @@ -38,6 +38,12 @@ </modules> </profile> <profile> + <id>build-debs</id> + <modules> + <module>packaging/docker/deb-docker</module> + </modules> + </profile> + <profile> <id>mpack</id> <activation> <activeByDefault>true</activeByDefault>
