Repository: incubator-samza-hello-samza Updated Branches: refs/heads/latest e7516ac22 -> 00e436644
SAMZA-451; remove vagrant Project: http://git-wip-us.apache.org/repos/asf/incubator-samza-hello-samza/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-samza-hello-samza/commit/00e43664 Tree: http://git-wip-us.apache.org/repos/asf/incubator-samza-hello-samza/tree/00e43664 Diff: http://git-wip-us.apache.org/repos/asf/incubator-samza-hello-samza/diff/00e43664 Branch: refs/heads/latest Commit: 00e436644f00840547bf8407d76f55bcac40d7ad Parents: e7516ac Author: Chris Riccomini <[email protected]> Authored: Thu Oct 30 15:03:43 2014 -0700 Committer: Chris Riccomini <[email protected]> Committed: Thu Oct 30 15:03:43 2014 -0700 ---------------------------------------------------------------------- README.md | 49 +---------------------------------------------- Vagrantfile | 35 --------------------------------- vagrant/bootstrap.sh | 46 -------------------------------------------- 3 files changed, 1 insertion(+), 129 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-samza-hello-samza/blob/00e43664/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index 3d5c323..a6066ea 100644 --- a/README.md +++ b/README.md @@ -5,53 +5,6 @@ Hello Samza is a starter project for [Apache Samza](http://samza.incubator.apach Please see [Hello Samza](http://samza.incubator.apache.org/startup/hello-samza/0.7.0/) to get started. -By default, Hello Samza uses a recent release of Samza from a Maven repository. If you want to use a custom -version of Samza, you can publish it to your local Maven repository in `$HOME/.m2` by running the following -in the Samza repository: - - ./gradlew publishToMavenLocal - -You can then use that version in Hello Samza by specifying the `samza.version` property when building -Hello Samza, for example: - - mvn package -Dsamza.version=0.8.0-SNAPSHOT - ### Pull requests and questions -Hello Samza is developed as part of the Apache Samza project. Please direct questions, improvements and -bug fixes there. Questions about Hello Samza are welcome on the dev list (details on the main -site above) and the Samza JIRA has a hello-samza component for filing tickets. - -### Using Vagrant - -If you'd like to use Vagrant to get up and running, follow these instructions. - -1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) -2) Install Virtual Box [https://www.virtualbox.org/](https://www.virtualbox.org/) - -Then once that is done (or if done already) clone this repository and boot the virtual machine up. - - cd hello-samza - vagrant up - -This will take ~ 10-15 minutes to install Kafka, Hadoop/YARN, Samza, configure everything together and launch the jobs. - -Once the VM is launched and you are back at a command prompt go into the virtual machine and see whats running. - - vagrant ssh - cd /vagrant - -The wikipedia-feed Samza job that is running is consuming a feed of real-time edits from Wikipedia, and producing them to a Kafka topic called "wikipedia-raw". You can view this in real-time by using the Kafka console consumer to view the topic. - - deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-raw - -The wikipedia-parser Samza job is then parsing the messages in wikipedia-raw, and extracting information about the size of the edit, who made the change, etc. It outputs these counts to the wikipedia-edits topic. - - deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-edits - -The wikipedia-stats Samza job reads messages from the wikipedia-edits topic, and calculates counts, every ten seconds, for all edits that were made during that window. It outputs these counts to the wikipedia-stats topic. - - deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-stats - -You can view the Samza jobs running in the YARN UI http://192.168.80.20:8088/cluster/apps too. -To see how this was setup and works look at `vagrant/bootstrap.sh` and [Hello Samza](http://samza.incubator.apache.org/). +[Hello Samza](http://samza.incubator.apache.org/startup/hello-samza/0.7.0/) is developed as part of the [Apache Samza](http://samza.incubator.apache.org) project. Please direct questions, improvements and bug fixes there. Questions about Hello Samza](http://samza.incubator.apache.org/startup/hello-samza/0.7.0/) are welcome on the [dev list](http://samza.incubator.apache.org/community/mailing-lists.html) and the [Samza JIRA](https://issues.apache.org/jira/browse/SAMZA) has a hello-samza component for filing tickets. http://git-wip-us.apache.org/repos/asf/incubator-samza-hello-samza/blob/00e43664/Vagrantfile ---------------------------------------------------------------------- diff --git a/Vagrantfile b/Vagrantfile deleted file mode 100644 index 5118bc6..0000000 --- a/Vagrantfile +++ /dev/null @@ -1,35 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -*- mode: ruby -*- -# vi: set ft=ruby : - -# Vagrantfile API/syntax version. Don't touch unless you know what you're doing! -VAGRANTFILE_API_VERSION = "2" - -Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| - config.vm.box = "precise64" - - # The url from where the 'config.vm.box' box will be fetched if it - # doesn't already exist on the user's system. - config.vm.box_url = "http://files.vagrantup.com/precise64.box"; - - config.vm.define "samza" do |samza| - samza.vm.network :private_network, ip: "192.168.80.20" - samza.vm.provider :virtualbox do |vb| - vb.customize ["modifyvm", :id, "--memory", "2048"] - end - samza.vm.provision "shell", path: "vagrant/bootstrap.sh", :args => "1" - end -end http://git-wip-us.apache.org/repos/asf/incubator-samza-hello-samza/blob/00e43664/vagrant/bootstrap.sh ---------------------------------------------------------------------- diff --git a/vagrant/bootstrap.sh b/vagrant/bootstrap.sh deleted file mode 100644 index 0631f78..0000000 --- a/vagrant/bootstrap.sh +++ /dev/null @@ -1,46 +0,0 @@ -#!/bin/bash -x -e -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -apt-get -y update -apt-get install -y software-properties-common python-software-properties -add-apt-repository -y ppa:webupd8team/java -apt-get -y update -/bin/echo debconf shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections -apt-get -y install oracle-java7-installer oracle-java7-set-default - -apt-get -y install git vim wget screen curl - -echo $'export JAVA_HOME=/usr\nexport PATH=/opt/apache/apache-maven-3.1.1/bin:$PATH' > /etc/profile.d/vagrant_samza.sh -. /etc/profile.d/vagrant_samza.sh - -cd /tmp -wget http://www.apache.org/dist/maven/binaries/apache-maven-3.1.1-bin.tar.gz -mkdir -p /opt/apache -cd /opt/apache/ -tar -xf /tmp/apache-maven-3.1.1-bin.tar.gz - -cd /vagrant -su vagrant -c "bin/grid bootstrap" - -su vagrant -c "/opt/apache/apache-maven-3.1.1/bin/mvn clean package" -su vagrant -c "mkdir -p deploy/samza" -su vagrant -c "tar -xf ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz -C deploy/samza" -su vagrant -c "deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://\$PWD/deploy/samza/config/wikipedia-feed.properties" -sleep 60 -su vagrant -c "deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://\$PWD/deploy/samza/config/wikipedia-parser.properties" -sleep 60 -su vagrant -c "deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://\$PWD/deploy/samza/config/wikipedia-stats.properties" -sleep 60
