SLIDER-977 Add ASL headers to all files. Of extreme importance, the license headers on app-packages/kafka/src/assembly/koya-app-package.xml and app-packages/kafka/src/assembly/koya-with-slider.xml were modified, not in content, but only formatting. These text of the license headers on these files was not changed.
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/58dfc2c5 Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/58dfc2c5 Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/58dfc2c5 Branch: refs/heads/feature/SLIDER-906_docker_support Commit: 58dfc2c54cc27b46ccc23b3a52ea47923c05898f Parents: 65e229e Author: Josh Elser <els...@apache.org> Authored: Tue Jan 19 22:41:27 2016 -0500 Committer: Josh Elser <els...@apache.org> Committed: Wed Jan 20 12:10:21 2016 -0500 ---------------------------------------------------------------------- app-packages/kafka/README.md | 29 +++++++++++---- app-packages/kafka/build.sh | 16 +++++++++ app-packages/kafka/configuration/server.xml | 16 +++++++++ app-packages/kafka/deploy.sh | 19 +++++++--- app-packages/kafka/flex.sh | 15 ++++++++ app-packages/kafka/metainfo-default.xml | 17 +++++++++ app-packages/kafka/package/scripts/kafka.py | 15 ++++++++ app-packages/kafka/package/scripts/params.py | 15 ++++++++ .../kafka/package/scripts/status_params.py | 16 ++++++++- app-packages/kafka/package/scripts/util.py | 18 +++++++++- app-packages/kafka/pom.xml | 38 ++++++++++++++------ .../kafka/src/assembly/koya-app-package.xml | 30 ++++++++-------- .../kafka/src/assembly/koya-with-slider.xml | 30 ++++++++-------- 13 files changed, 219 insertions(+), 55 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58dfc2c5/app-packages/kafka/README.md ---------------------------------------------------------------------- diff --git a/app-packages/kafka/README.md b/app-packages/kafka/README.md index 97ddef4..34f0d04 100644 --- a/app-packages/kafka/README.md +++ b/app-packages/kafka/README.md @@ -1,3 +1,20 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + Kafka On YARN (KOYA) ==================== @@ -17,7 +34,7 @@ Kafka as YARN application using Slider ### Build -Checkout Slider code (https://github.com/apache/incubator-slider) +Checkout Slider code (https://github.com/apache/incubator-slider) ```sh git clone g...@github.com:apache/incubator-slider.git git checkout -b slider-0.80.0-incubating remotes/origin/releases/slider-0.80.0-incubating @@ -44,14 +61,14 @@ Artifacts: To use the archive with embedded Slider, copy it to the machine from which you launch YARN applications (Hadoop client, gateway or edge node). Extract the file and configure Slider: -If the environment variables `HADOOP_CONF_DIR` or `JAVA_HOME` are not already defined through your Hadoop installation, you can export them in `slider-0.80.0-incubating/conf/slider-env.sh` +If the environment variables `HADOOP_CONF_DIR` or `JAVA_HOME` are not already defined through your Hadoop installation, you can export them in `slider-0.80.0-incubating/conf/slider-env.sh` Example for CDH 5.4: -``` +``` export HADOOP_CONF_DIR=/etc/hadoop/conf export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera ``` -If the registry ZooKeeper quorum was not already configured through Hadoop, modify `slider-0.80.0-incubating/conf/slider-client.xml`: +If the registry ZooKeeper quorum was not already configured through Hadoop, modify `slider-0.80.0-incubating/conf/slider-client.xml`: ``` <property> <name>hadoop.registry.zk.quorum</name> @@ -72,9 +89,9 @@ If you use the full archive, the configuration file templates are already in you Extract the packaged configuration files you are going to customize: ``` -unzip koya-slider-package-0.1.zip appConfig.json resources.json +unzip koya-slider-package-0.1.zip appConfig.json resources.json ``` -Adjust following properties in the global section: +Adjust following properties in the global section: ``` "application.def": "koya-slider-package-0.1.zip", "site.global.xmx_val": "256m", http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58dfc2c5/app-packages/kafka/build.sh ---------------------------------------------------------------------- diff --git a/app-packages/kafka/build.sh b/app-packages/kafka/build.sh index 7951f9e..35eeeba 100755 --- a/app-packages/kafka/build.sh +++ b/app-packages/kafka/build.sh @@ -1,5 +1,21 @@ #! /bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + wget http://apache.websitebeheerjd.nl/kafka/0.8.2.1/kafka_2.10-0.8.2.1.tgz -O $HOME/kafka_2.10-0.8.2.1.tgz mvn clean install -DskipTests -Dkafka.src=$HOME/kafka_2.10-0.8.2.1.tgz -Dkafka.version=kafka_2.10-0.8.2.1 unzip -o target/koya-slider-package-0.1.zip appConfig.json http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58dfc2c5/app-packages/kafka/configuration/server.xml ---------------------------------------------------------------------- diff --git a/app-packages/kafka/configuration/server.xml b/app-packages/kafka/configuration/server.xml index 68b36c9..b22db45 100644 --- a/app-packages/kafka/configuration/server.xml +++ b/app-packages/kafka/configuration/server.xml @@ -1,5 +1,21 @@ <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> <configuration> <property> <name>zookeeper.connect</name> http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58dfc2c5/app-packages/kafka/deploy.sh ---------------------------------------------------------------------- diff --git a/app-packages/kafka/deploy.sh b/app-packages/kafka/deploy.sh index 1f8f473..9b69c90 100755 --- a/app-packages/kafka/deploy.sh +++ b/app-packages/kafka/deploy.sh @@ -1,11 +1,22 @@ #! /bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + APPNAME=${1:-koya} slider install-package --replacepkg --name KOYA --package target/koya-slider-package-0.1.zip slider stop $APPNAME slider destroy $APPNAME slider create $APPNAME --filesystem hdfs://root --queue dev --template appConfig.json --resources resources-default.json - - - - http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58dfc2c5/app-packages/kafka/flex.sh ---------------------------------------------------------------------- diff --git a/app-packages/kafka/flex.sh b/app-packages/kafka/flex.sh index 20a1003..36c769b 100755 --- a/app-packages/kafka/flex.sh +++ b/app-packages/kafka/flex.sh @@ -1,5 +1,20 @@ #! /bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + APPNAME=${1:-koya} APPNUM=${2:-12} slider flex $APPNAME --component BROKER0 $APPNUM --filesystem hdfs://root http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58dfc2c5/app-packages/kafka/metainfo-default.xml ---------------------------------------------------------------------- diff --git a/app-packages/kafka/metainfo-default.xml b/app-packages/kafka/metainfo-default.xml index f1eec0a..05dd1f6 100644 --- a/app-packages/kafka/metainfo-default.xml +++ b/app-packages/kafka/metainfo-default.xml @@ -1,3 +1,20 @@ +<?xml version="1.0"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> <metainfo> <schemaVersion>2.0</schemaVersion> <application> http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58dfc2c5/app-packages/kafka/package/scripts/kafka.py ---------------------------------------------------------------------- diff --git a/app-packages/kafka/package/scripts/kafka.py b/app-packages/kafka/package/scripts/kafka.py index 38b12a3..325c65c 100644 --- a/app-packages/kafka/package/scripts/kafka.py +++ b/app-packages/kafka/package/scripts/kafka.py @@ -1,3 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + import logging import sys import os http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58dfc2c5/app-packages/kafka/package/scripts/params.py ---------------------------------------------------------------------- diff --git a/app-packages/kafka/package/scripts/params.py b/app-packages/kafka/package/scripts/params.py index de3994e..1bc3f2f 100644 --- a/app-packages/kafka/package/scripts/params.py +++ b/app-packages/kafka/package/scripts/params.py @@ -1,3 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + from resource_management import * config = Script.get_config() http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58dfc2c5/app-packages/kafka/package/scripts/status_params.py ---------------------------------------------------------------------- diff --git a/app-packages/kafka/package/scripts/status_params.py b/app-packages/kafka/package/scripts/status_params.py index cc46d5b..7ae91bd 100644 --- a/app-packages/kafka/package/scripts/status_params.py +++ b/app-packages/kafka/package/scripts/status_params.py @@ -1,3 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + from resource_management import * config = Script.get_config() @@ -6,4 +21,3 @@ app_root = config['configurations']['global']['app_root'] java64_home = config['hostLevelParams']['java_home'] app_user = config['configurations']['global']['app_user'] pid_file = config['configurations']['global']['pid_file'] - http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58dfc2c5/app-packages/kafka/package/scripts/util.py ---------------------------------------------------------------------- diff --git a/app-packages/kafka/package/scripts/util.py b/app-packages/kafka/package/scripts/util.py index ca4cb41..996a897 100644 --- a/app-packages/kafka/package/scripts/util.py +++ b/app-packages/kafka/package/scripts/util.py @@ -1,3 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + import re from os import fsync @@ -10,7 +26,7 @@ def updating(filename,dico): return str(dic[mat.group(2)]).join(mat.group(1,3)) with open(filename,'rb') as f: - content = f.read() + content = f.read() with open(filename,'wb') as f: f.write(pat.sub(jojo,content)) http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58dfc2c5/app-packages/kafka/pom.xml ---------------------------------------------------------------------- diff --git a/app-packages/kafka/pom.xml b/app-packages/kafka/pom.xml index 267eb04..fbb9db8 100644 --- a/app-packages/kafka/pom.xml +++ b/app-packages/kafka/pom.xml @@ -1,15 +1,32 @@ <?xml version="1.0"?> <project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";> - <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + <!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + <parent> + <groupId>org.apache.slider.packages</groupId> + <artifactId>slider-app-packages</artifactId> + <version>0.90.0-incubating-SNAPSHOT</version> + <relativePath>../pom.xml</relativePath> + </parent> <modelVersion>4.0.0</modelVersion> - <groupId>com.datatorrent</groupId> - <version>0.1</version> - <artifactId>koya-slider-package</artifactId> + <artifactId>kafka-app-package</artifactId> <packaging>jar</packaging> - <name>koya-slider-package</name> - <description>A package of koya on slider</description> + <name>Slider Kafka App Package</name> + <description>Slider Kafka App Package</description> <properties> <slider.version>0.90.0-incubating-SNAPSHOT</slider.version> @@ -22,7 +39,7 @@ <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> - <version>2.5.4</version> + <version>>${maven-assembly-plugin.version}</version> <executions> <execution> <id>app-package</id> @@ -35,12 +52,12 @@ <appendAssemblyId>false</appendAssemblyId> <finalName>${app.package.name}</finalName> </configuration> - </execution> + </execution> <execution> <id>package-with-slider</id> - <phase>package</phase> + <phase>package</phase> <goals> - <goal>single</goal> + <goal>single</goal> </goals> <configuration> <descriptor>src/assembly/koya-with-slider.xml</descriptor> @@ -66,4 +83,3 @@ </project> - http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58dfc2c5/app-packages/kafka/src/assembly/koya-app-package.xml ---------------------------------------------------------------------- diff --git a/app-packages/kafka/src/assembly/koya-app-package.xml b/app-packages/kafka/src/assembly/koya-app-package.xml index dd8bd5c..9d984c1 100644 --- a/app-packages/kafka/src/assembly/koya-app-package.xml +++ b/app-packages/kafka/src/assembly/koya-app-package.xml @@ -1,21 +1,19 @@ <!-- - ~ Licensed to the Apache Software Foundation (ASF) under one - ~ or more contributor license agreements. See the NOTICE file - ~ distributed with this work for additional information - ~ regarding copyright ownership. The ASF licenses this file - ~ to you under the Apache License, Version 2.0 (the - ~ "License"); you may not use this file except in compliance - ~ with the License. You may obtain a copy of the License at - ~ - ~ http://www.apache.org/licenses/LICENSE-2.0 - ~ - ~ Unless required by applicable law or agreed to in writing, software - ~ distributed under the License is distributed on an "AS IS" BASIS, - ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - ~ See the License for the specific language governing permissions and - ~ limitations under the License. - --> + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> <assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"; http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/58dfc2c5/app-packages/kafka/src/assembly/koya-with-slider.xml ---------------------------------------------------------------------- diff --git a/app-packages/kafka/src/assembly/koya-with-slider.xml b/app-packages/kafka/src/assembly/koya-with-slider.xml index 0f64927..98d730c 100644 --- a/app-packages/kafka/src/assembly/koya-with-slider.xml +++ b/app-packages/kafka/src/assembly/koya-with-slider.xml @@ -1,21 +1,19 @@ <!-- - ~ Licensed to the Apache Software Foundation (ASF) under one - ~ or more contributor license agreements. See the NOTICE file - ~ distributed with this work for additional information - ~ regarding copyright ownership. The ASF licenses this file - ~ to you under the Apache License, Version 2.0 (the - ~ "License"); you may not use this file except in compliance - ~ with the License. You may obtain a copy of the License at - ~ - ~ http://www.apache.org/licenses/LICENSE-2.0 - ~ - ~ Unless required by applicable law or agreed to in writing, software - ~ distributed under the License is distributed on an "AS IS" BASIS, - ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - ~ See the License for the specific language governing permissions and - ~ limitations under the License. - --> + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> <assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0";
