SLIDER-977 Address a bunch of consistency issues in the kafka-app-package Match the "naming" (KOYA can be the short name, but slider-kafka-app-package would be the proper name). Updated docs where I saw necessary. Make sure we include the version string in all artifacts. Dynamically pull down a the kafka tarball in the build.
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/7bb6116b Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/7bb6116b Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/7bb6116b Branch: refs/heads/develop Commit: 7bb6116bef7bc6ef7c7ff3ccbf024334d6e110cf Parents: 58dfc2c Author: Josh Elser <els...@apache.org> Authored: Tue Jan 19 23:14:49 2016 -0500 Committer: Josh Elser <els...@apache.org> Committed: Wed Jan 20 12:10:21 2016 -0500 ---------------------------------------------------------------------- app-packages/kafka/README.md | 10 ++--- app-packages/kafka/appConfig-default.json | 2 +- app-packages/kafka/build.sh | 2 +- app-packages/kafka/deploy.sh | 2 +- app-packages/kafka/pom.xml | 43 ++++++++++++++++++-- .../kafka/src/assembly/koya-with-slider.xml | 2 +- app-packages/pom.xml | 2 + pom.xml | 1 + 8 files changed, 51 insertions(+), 13 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/7bb6116b/app-packages/kafka/README.md ---------------------------------------------------------------------- diff --git a/app-packages/kafka/README.md b/app-packages/kafka/README.md index 34f0d04..0a3de5a 100644 --- a/app-packages/kafka/README.md +++ b/app-packages/kafka/README.md @@ -53,7 +53,7 @@ mvn clean install -DskipTests -Dkafka.src=path/to/kafka_2.10-0.8.1.1.tgz -Dkafka Artifacts: - Archive with embedded Slider: __`target/koya-with-slider.zip`__ - - Separate Slider application package: __`target/koya-slider-package-0.1.zip`__ + - Separate Slider application package: __`target/slider-kafka-app-package-0.90.0-incubating-SNAPSHOT.zip`__ ###Installation @@ -89,11 +89,11 @@ If you use the full archive, the configuration file templates are already in you Extract the packaged configuration files you are going to customize: ``` -unzip koya-slider-package-0.1.zip appConfig.json resources.json +unzip slider-kafka-app-package-0.90.0-incubating-SNAPSHOT.zip appConfig.json resources.json ``` Adjust following properties in the global section: ``` - "application.def": "koya-slider-package-0.1.zip", + "application.def": "slider-kafka-app-package-0.90.0-incubating-SNAPSHOT.zip", "site.global.xmx_val": "256m", "site.global.xms_val": "128m", "site.broker.zookeeper.connect": "${ZK_HOST}" @@ -119,9 +119,9 @@ More information about the application configuration can be found [here](http:// The Slider application package needs to be copied to the HDFS location that was specified as application.def in appConfig.json: ``` -hdfs dfs -copyFromLocal koya-slider-package-0.1.zip /path/in/appConfig +hdfs dfs -copyFromLocal slider-kafka-app-package-0.90.0-incubating-SNAPSHOT.zip /path/in/appConfig ``` Now the KOYA cluster can be deployed and launched: ``` -slider-0.80.0-incubating/bin/slider create koya --template ~/koya/appConfig.json --resources ~/koya/resources.json +slider-0.90.0-incubating/bin/slider create koya --template ~/koya/appConfig.json --resources ~/koya/resources.json ``` http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/7bb6116b/app-packages/kafka/appConfig-default.json ---------------------------------------------------------------------- diff --git a/app-packages/kafka/appConfig-default.json b/app-packages/kafka/appConfig-default.json index 9a0be4c..d7b6d6f 100644 --- a/app-packages/kafka/appConfig-default.json +++ b/app-packages/kafka/appConfig-default.json @@ -3,7 +3,7 @@ "metadata": { }, "global": { - "application.def": ".slider/package/KOYA/koya-slider-package-${project.version}.zip", + "application.def": ".slider/package/KOYA/slider-kafka-app-package-${project.version}.zip", "java_home": "${JAVA_HOME}", "system_configs": "broker", http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/7bb6116b/app-packages/kafka/build.sh ---------------------------------------------------------------------- diff --git a/app-packages/kafka/build.sh b/app-packages/kafka/build.sh index 35eeeba..cc5207b 100755 --- a/app-packages/kafka/build.sh +++ b/app-packages/kafka/build.sh @@ -18,4 +18,4 @@ wget http://apache.websitebeheerjd.nl/kafka/0.8.2.1/kafka_2.10-0.8.2.1.tgz -O $HOME/kafka_2.10-0.8.2.1.tgz mvn clean install -DskipTests -Dkafka.src=$HOME/kafka_2.10-0.8.2.1.tgz -Dkafka.version=kafka_2.10-0.8.2.1 -unzip -o target/koya-slider-package-0.1.zip appConfig.json +unzip -o target/slider-kafka-app-package-0.90.0-incubating-SNAPSHOT.zip appConfig.json http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/7bb6116b/app-packages/kafka/deploy.sh ---------------------------------------------------------------------- diff --git a/app-packages/kafka/deploy.sh b/app-packages/kafka/deploy.sh index 9b69c90..11e32f3 100755 --- a/app-packages/kafka/deploy.sh +++ b/app-packages/kafka/deploy.sh @@ -16,7 +16,7 @@ # limitations under the License. APPNAME=${1:-koya} -slider install-package --replacepkg --name KOYA --package target/koya-slider-package-0.1.zip +slider install-package --replacepkg --name KOYA --package target/slider-kafka-app-package-0.90.0-incubating-SNAPSHOT.zip slider stop $APPNAME slider destroy $APPNAME slider create $APPNAME --filesystem hdfs://root --queue dev --template appConfig.json --resources resources-default.json http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/7bb6116b/app-packages/kafka/pom.xml ---------------------------------------------------------------------- diff --git a/app-packages/kafka/pom.xml b/app-packages/kafka/pom.xml index fbb9db8..0a5f1e1 100644 --- a/app-packages/kafka/pom.xml +++ b/app-packages/kafka/pom.xml @@ -23,14 +23,14 @@ <relativePath>../pom.xml</relativePath> </parent> <modelVersion>4.0.0</modelVersion> - <artifactId>kafka-app-package</artifactId> + <artifactId>slider-kafka-app-package</artifactId> <packaging>jar</packaging> <name>Slider Kafka App Package</name> <description>Slider Kafka App Package</description> <properties> <slider.version>0.90.0-incubating-SNAPSHOT</slider.version> - <kafka.src>${user.home}/kafka_2.10-0.8.2.1.tgz</kafka.src> + <kafka.src>${project.build.directory}/kafka_2.10-0.8.2.1.tgz</kafka.src> <kafka.version>kafka_2.10-0.8.2.1</kafka.version> </properties> @@ -39,7 +39,7 @@ <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> - <version>>${maven-assembly-plugin.version}</version> + <version>${maven-assembly-plugin.version}</version> <executions> <execution> <id>app-package</id> @@ -62,13 +62,48 @@ <configuration> <descriptor>src/assembly/koya-with-slider.xml</descriptor> <appendAssemblyId>false</appendAssemblyId> - <finalName>koya-with-slider</finalName> + <finalName>slider-kafka-app-package-with-slider-${project.version}</finalName> <attach>false</attach> </configuration> </execution> </executions> </plugin> + <plugin> + <artifactId>maven-compiler-plugin</artifactId> + <version>${maven-compiler-plugin.version}</version> + <dependencies> + <dependency> + <groupId>org.codehaus.groovy</groupId> + <artifactId>groovy-eclipse-compiler</artifactId> + <version>${groovy-eclipse-compiler.version}</version> + </dependency> + <dependency> + <groupId>org.codehaus.groovy</groupId> + <artifactId>groovy-eclipse-batch</artifactId> + <version>${groovy-eclipse-batch.version}</version> + </dependency> + </dependencies> + </plugin> + <plugin> + <groupId>org.codehaus.mojo</groupId> + <artifactId>wagon-maven-plugin</artifactId> + <version>${wagon-maven-plugin.version}</version> + <executions> + <execution> + <id>download-kafka</id> + <phase>prepare-package</phase> + <goals> + <goal>download-single</goal> + </goals> + <configuration> + <url>https://dist.apache.org</url> + <fromFile>repos/dist/release/kafka/0.8.2.1/kafka_2.10-0.8.2.1.tgz</fromFile> + <toDir>${project.build.directory}</toDir> + </configuration> + </execution> + </executions> + </plugin> </plugins> </build> http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/7bb6116b/app-packages/kafka/src/assembly/koya-with-slider.xml ---------------------------------------------------------------------- diff --git a/app-packages/kafka/src/assembly/koya-with-slider.xml b/app-packages/kafka/src/assembly/koya-with-slider.xml index 98d730c..a5c6384 100644 --- a/app-packages/kafka/src/assembly/koya-with-slider.xml +++ b/app-packages/kafka/src/assembly/koya-with-slider.xml @@ -27,7 +27,7 @@ <files> <file> - <source>target/koya-slider-package-${project.version}.zip</source> + <source>target/slider-kafka-app-package-${project.version}.zip</source> <outputDirectory>/</outputDirectory> <fileMode>0755</fileMode> </file> http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/7bb6116b/app-packages/pom.xml ---------------------------------------------------------------------- diff --git a/app-packages/pom.xml b/app-packages/pom.xml index a286d0c..aa9632b 100644 --- a/app-packages/pom.xml +++ b/app-packages/pom.xml @@ -52,6 +52,7 @@ <modules> <module>accumulo</module> <module>hbase</module> + <module>kafka</module> <module>storm</module> <module>tomcat</module> </modules> @@ -83,6 +84,7 @@ <modules> <module>accumulo</module> <module>hbase</module> + <module>kafka</module> <module>storm</module> <module>hbase-win</module> <module>storm-win</module> http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/7bb6116b/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index d39a4e4..5e8cbc2 100644 --- a/pom.xml +++ b/pom.xml @@ -223,6 +223,7 @@ <maven-failsafe-plugin.version>${maven-surefire-plugin.version}</maven-failsafe-plugin.version> <maven-verifier-plugin.version>1.0</maven-verifier-plugin.version> <apache-rat-plugin.version>0.11</apache-rat-plugin.version> + <wagon-maven-plugin.version>1.0</wagon-maven-plugin.version> <!-- build options--> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
