This is an automated email from the ASF dual-hosted git repository.
bharathkk pushed a commit to branch 1.5.0
in repository https://gitbox.apache.org/repos/asf/samza.git
The following commit(s) were added to refs/heads/1.5.0 by this push:
new 05b0800 SAMZA-2442: Update doc/tutorials from config factory to
config loader
05b0800 is described below
commit 05b0800643fdf2b4bfa0b9f48b13f9b3be81a10b
Author: Ke Wu <[email protected]>
AuthorDate: Wed Jun 3 14:36:00 2020 -0700
SAMZA-2442: Update doc/tutorials from config factory to config loader
Update documentations and left over scripts from config factory to config
loader.
Tests:
1. Test samza rest following instructions on
https://samza.apache.org/learn/documentation/latest/rest/overview.html
2.
API Changes:
None
Upgrade Instructions:
None
Usage Instructions:
None
Author: Ke Wu <[email protected]>
Reviewers: Cameron Lee <[email protected]>
Closes #1332 from kw2542/SAMZA-2442
(cherry picked from commit fbe8349714d5ba58b169384d11c8f856613d8919)
Signed-off-by: mynameborat <[email protected]>
---
bin/setup-int-test.sh | 2 +-
.../versioned/deployment/standalone.md | 2 +-
.../documentation/versioned/deployment/yarn.md | 2 +-
.../documentation/versioned/jobs/job-runner.md | 60 ----------------------
.../learn/documentation/versioned/rest/overview.md | 4 +-
.../tutorials/versioned/deploy-samza-to-CDH.md | 2 +-
.../versioned/hello-samza-high-level-yarn.md | 4 +-
.../tutorials/versioned/remote-debugging-samza.md | 2 +-
.../versioned/run-hello-samza-without-internet.md | 4 +-
.../tutorials/versioned/run-in-multi-node-yarn.md | 2 +-
.../versioned/samza-rest-getting-started.md | 6 +--
docs/learn/tutorials/versioned/samza-sql.md | 4 +-
docs/startup/hello-samza/versioned/index.md | 12 ++---
docs/startup/quick-start/versioned/samza-sql.md | 2 +-
docs/startup/quick-start/versioned/samza.md | 2 +-
.../stream/CoordinatorStreamWriter.java | 4 +-
samza-rest/src/main/bash/run-samza-rest-service.sh | 2 +-
.../org/apache/samza/rest/SamzaRestService.java | 3 +-
.../samza/rest/proxy/job/SimpleYarnJobProxy.java | 7 ++-
samza-test/src/main/python/configs/tests.json | 2 +-
samza-test/src/main/python/deployment.py | 2 +-
.../src/main/python/samza_job_yarn_deployer.py | 24 ++++-----
.../samza/validation/YarnJobValidationTool.java | 2 +-
23 files changed, 47 insertions(+), 109 deletions(-)
diff --git a/bin/setup-int-test.sh b/bin/setup-int-test.sh
index 130be5d..d0fa832 100755
--- a/bin/setup-int-test.sh
+++ b/bin/setup-int-test.sh
@@ -43,7 +43,7 @@ $KAFKA_DIR/bin/kafka-topics.sh --zookeeper localhost:2181
--partitions 1 --repli
# Start the jobs
for job in checker joiner emitter watcher
do
- $SAMZA_DIR/bin/run-job.sh --config
job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory
--config job.config.loader.properties.path=$SAMZA_DIR/config/join/$job.samza
--config job.foo=$job
+ $SAMZA_DIR/bin/run-app.sh --config-path=$SAMZA_DIR/config/join/$job.samza
--config job.foo=$job
done
diff --git a/docs/learn/documentation/versioned/deployment/standalone.md
b/docs/learn/documentation/versioned/deployment/standalone.md
index 6b2d168..115e3f3 100644
--- a/docs/learn/documentation/versioned/deployment/standalone.md
+++ b/docs/learn/documentation/versioned/deployment/standalone.md
@@ -72,7 +72,7 @@ tar -xvf ./target/hello-samza-1.1.0-dist.tar.gz -C
deploy/samza
We are ready to run the example application
[WikipediaZkLocalApplication](https://github.com/apache/samza-hello-samza/blob/master/src/main/java/samza/examples/wikipedia/application/WikipediaZkLocalApplication.java).
This application reads messages from the wikipedia-edits topic, and calculates
counts, every ten seconds, for all edits that were made during that window. It
emits these results to another topic named `wikipedia-stats`.
```bash
-./deploy/samza/bin/run-class.sh
samza.examples.wikipedia.application.WikipediaZkLocalApplication
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/wikipedia-application-local-runner.properties
+./deploy/samza/bin/run-class.sh
samza.examples.wikipedia.application.WikipediaZkLocalApplication --config
job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory
--config
job.config.loader.properties.path=$PWD/deploy/samza/config/wikipedia-application-local-runner.properties
```
You can run the above command again to spin up a new instance of your
application.
diff --git a/docs/learn/documentation/versioned/deployment/yarn.md
b/docs/learn/documentation/versioned/deployment/yarn.md
index 3a46cea..36f88c0 100644
--- a/docs/learn/documentation/versioned/deployment/yarn.md
+++ b/docs/learn/documentation/versioned/deployment/yarn.md
@@ -97,7 +97,7 @@ samza-job-name-folder
Once the archive is built, the `run-app.sh` script can be used to submit the
application to YARN's resource manager. The script takes 2 CLI parameters - the
config factory and the config file for the application. As an example, lets run
our
[FilterExample](https://github.com/apache/samza-hello-samza/blob/latest/src/main/java/samza/examples/cookbook/FilterExample.java)
on YARN as follows:
```bash
-$ ./deploy/samza/bin/run-app.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path ./deploy/samza/config/filter-example.properties
+$ ./deploy/samza/bin/run-app.sh
--config-path=./deploy/samza/config/filter-example.properties
```
Congratulations, you've successfully submitted your first job to YARN! You can
view the YARN Web UI to view its status.
diff --git a/docs/learn/documentation/versioned/jobs/job-runner.md
b/docs/learn/documentation/versioned/jobs/job-runner.md
deleted file mode 100644
index 5f0607f..0000000
--- a/docs/learn/documentation/versioned/jobs/job-runner.md
+++ /dev/null
@@ -1,60 +0,0 @@
----
-layout: page
-title: JobRunner
----
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-Samza jobs are started using a script called run-job.sh.
-
-{% highlight bash %}
-samza-example/target/bin/run-job.sh \
- --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory \
- --config-path=file://$PWD/config/hello-world.properties
-{% endhighlight %}
-
-You provide two parameters to the run-job.sh script. One is the config
location, and the other is a factory class that is used to read your
configuration file. The run-job.sh script is actually executing a Samza class
called JobRunner. The JobRunner uses your ConfigFactory to get a Config object
from the config path.
-
-{% highlight java %}
-public interface ConfigFactory {
- Config getConfig(URI configUri);
-}
-{% endhighlight %}
-
-The Config object is just a wrapper around Map<String, String>, with some nice
helper methods. Out of the box, Samza ships with the PropertiesConfigFactory,
but developers can implement any kind of ConfigFactory they wish.
-
-Once the JobRunner gets your configuration, it gives your configuration to the
StreamJobFactory class defined by the "job.factory" property. Samza ships with
three job factory implementations: ThreadJobFactory, ProcessJobFactory and
YarnJobFactory. The StreamJobFactory's responsibility is to give the JobRunner
a job that it can run.
-
-{% highlight java %}
-public interface StreamJob {
- StreamJob submit();
-
- StreamJob kill();
-
- ApplicationStatus waitForFinish(long timeoutMs);
-
- ApplicationStatus waitForStatus(ApplicationStatus status, long timeoutMs);
-
- ApplicationStatus getStatus();
-}
-{% endhighlight %}
-
-Once the JobRunner gets a job, it calls submit() on the job. This method is
what tells the StreamJob implementation to start the SamzaContainer. In the
case of LocalJobRunner, it uses a run-container.sh script to execute the
SamzaContainer in a separate process, which will start one SamzaContainer
locally on the machine that you ran run-job.sh on.
-
-This flow differs slightly when you use YARN, but we'll get to that later.
-
-## [Configuration »](configuration.html)
diff --git a/docs/learn/documentation/versioned/rest/overview.md
b/docs/learn/documentation/versioned/rest/overview.md
index c382f03..2872bd0 100644
--- a/docs/learn/documentation/versioned/rest/overview.md
+++ b/docs/learn/documentation/versioned/rest/overview.md
@@ -37,8 +37,8 @@ Deploying the service is very similar to running a Samza job.
First build the ta
Then from the extracted location, run the service using:
{% highlight bash %}
samza-example/target/bin/run-samza-rest-service.sh \
- --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory \
- --config-path=file://$PWD/config/samza-rest.properties
+ --config
job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory
\
+ --config job.config.loader.properties.path=$PWD/config/samza-rest.properties
{% endhighlight %}
The two config parameters have the same purpose as they do for
[run-job.sh](../jobs/job-runner.html).
diff --git a/docs/learn/tutorials/versioned/deploy-samza-to-CDH.md
b/docs/learn/tutorials/versioned/deploy-samza-to-CDH.md
index e3b920b..f215a6a 100644
--- a/docs/learn/tutorials/versioned/deploy-samza-to-CDH.md
+++ b/docs/learn/tutorials/versioned/deploy-samza-to-CDH.md
@@ -66,5 +66,5 @@ export HADOOP_CONF_DIR=/etc/hadoop/conf
### Run Samza Job
{% highlight bash %}
-bin/run-job.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/config/wikipedia-parser.properties
+bin/run-app.sh --config-path=$PWD/config/wikipedia-parser.properties
{% endhighlight %}
diff --git a/docs/learn/tutorials/versioned/hello-samza-high-level-yarn.md
b/docs/learn/tutorials/versioned/hello-samza-high-level-yarn.md
index 69abf68..b0fc9fe 100644
--- a/docs/learn/tutorials/versioned/hello-samza-high-level-yarn.md
+++ b/docs/learn/tutorials/versioned/hello-samza-high-level-yarn.md
@@ -71,7 +71,7 @@ tar -xvf ./target/hello-samza-1.4.0-SNAPSHOT-dist.tar.gz -C
deploy/samza
After you've built your Samza package, you can start the app on the grid using
the run-app.sh script.
{% highlight bash %}
-./deploy/samza/bin/run-app.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/wikipedia-application.properties
+./deploy/samza/bin/run-app.sh
--config-path=$PWD/deploy/samza/config/wikipedia-application.properties
{% endhighlight %}
The app will do all of the following:
@@ -115,7 +115,7 @@ This plan will make more sense after the [code
walkthrough](hello-samza-high-lev
To shutdown the app, use the same _run-app.sh_ script with an extra
_--operation=kill_ argument
{% highlight bash %}
-./deploy/samza/bin/run-app.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/wikipedia-application.properties
--operation=kill
+./deploy/samza/bin/run-app.sh
--config-path=$PWD/deploy/samza/config/wikipedia-application.properties
--operation=kill
{% endhighlight %}
After you're done, you can clean everything up using the same grid script.
diff --git a/docs/learn/tutorials/versioned/remote-debugging-samza.md
b/docs/learn/tutorials/versioned/remote-debugging-samza.md
index e0c7445..88c85fb 100644
--- a/docs/learn/tutorials/versioned/remote-debugging-samza.md
+++ b/docs/learn/tutorials/versioned/remote-debugging-samza.md
@@ -84,7 +84,7 @@ Once the grid starts, you can start the wikipedia-feed Samza
job.
mvn clean package
mkdir -p deploy/samza
tar -xvf ./target/hello-samza-1.1.0-dist.tar.gz -C deploy/samza
-deploy/samza/bin/run-job.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
+deploy/samza/bin/run-app.sh
--config-path=$PWD/deploy/samza/config/wikipedia-feed.properties
{% endhighlight %}
When the wikipedia-feed job starts up, a single Samza container will be
created to process all incoming messages. This is the container that we'll want
to connect to from the remote debugger.
diff --git a/docs/learn/tutorials/versioned/run-hello-samza-without-internet.md
b/docs/learn/tutorials/versioned/run-hello-samza-without-internet.md
index e276cdb..487bd42 100644
--- a/docs/learn/tutorials/versioned/run-hello-samza-without-internet.md
+++ b/docs/learn/tutorials/versioned/run-hello-samza-without-internet.md
@@ -48,7 +48,7 @@ Otherwise, you may have the connection problem.
We provide an alternative to get wikipedia feed data. Instead of running
{% highlight bash %}
-deploy/samza/bin/run-job.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
+deploy/samza/bin/run-app.sh
--config-path=$PWD/deploy/samza/config/wikipedia-feed.properties
{% endhighlight %}
You will run
@@ -70,7 +70,7 @@ Now you can go back to Generate Wikipedia Statistics section
in [Hello Samza](..
The goal of
{% highlight bash %}
-deploy/samza/bin/run-job.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
+deploy/samza/bin/run-app.sh
--config-path=$PWD/deploy/samza/config/wikipedia-feed.properties
{% endhighlight %}
is to deploy a Samza job which listens to wikipedia API, receives the feed in
realtime and produces the feed to the Kafka topic wikipedia-raw. The
alternative in this tutorial is reading local wikipedia feed in an infinite
loop and producing the data to Kafka wikipedia-raw. The follow-up job,
wikipedia-parser is getting data from Kafka topic wikipedia-raw, so as long as
we have correct data in Kafka topic wikipedia-raw, we are fine. All Samza jobs
are connected by the Kafka and do not de [...]
diff --git a/docs/learn/tutorials/versioned/run-in-multi-node-yarn.md
b/docs/learn/tutorials/versioned/run-in-multi-node-yarn.md
index 837f6ed..e4745ce 100644
--- a/docs/learn/tutorials/versioned/run-in-multi-node-yarn.md
+++ b/docs/learn/tutorials/versioned/run-in-multi-node-yarn.md
@@ -167,7 +167,7 @@ cd /tmp/hello-samza && python -m SimpleHTTPServer
Go back to the original terminal (not the one running the HTTP server):
{% highlight bash %}
-deploy/samza/bin/run-job.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
+deploy/samza/bin/run-app.sh
--config-path=$PWD/deploy/samza/config/wikipedia-feed.properties
{% endhighlight %}
Go to http://yourHostname:8088 and find the wikipedia-feed job. Click on the
ApplicationMaster link to see that it's running.
diff --git a/docs/learn/tutorials/versioned/samza-rest-getting-started.md
b/docs/learn/tutorials/versioned/samza-rest-getting-started.md
index bd8137a..d1a8b4b 100644
--- a/docs/learn/tutorials/versioned/samza-rest-getting-started.md
+++ b/docs/learn/tutorials/versioned/samza-rest-getting-started.md
@@ -48,7 +48,7 @@ Run the following commands:
{% highlight bash %}
cd samza-rest/build/distributions/
mkdir -p deploy/samza-rest
-tar -xvf ./samza-rest-1.4.0-SNAPSHOT.tgz -C deploy/samza-rest
+tar -xvf ./samza-rest_2.11-1.4.0-SNAPSHOT.tgz -C deploy/samza-rest
{% endhighlight %}
#### Configure the Installations Path
@@ -71,8 +71,8 @@ To deploy the service, run the run-samza-rest-service.sh
script from the extract
{% highlight bash %}
cd deploy/samza-rest
./bin/run-samza-rest-service.sh \
- --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory \
- --config-path=file://$PWD/config/samza-rest.properties
+ --config
job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory
\
+ --config job.config.loader.properties.path=$PWD/config/samza-rest.properties
{% endhighlight %}
You provide two parameters to the run-samza-rest-service.sh script. One is the
config location, and the other, optional, parameter is a factory class that is
used to read your configuration file. The SamzaRestService uses your
ConfigFactory to get a Config object from the config path. The ConfigFactory is
covered in more detail on the [Job Runner
page](../../documentation/{{site.version}}/jobs/job-runner.html). The
run-samza-rest-service.sh script will block until the SamzaRestService te [...]
diff --git a/docs/learn/tutorials/versioned/samza-sql.md
b/docs/learn/tutorials/versioned/samza-sql.md
index f905d4a..98a86b4 100644
--- a/docs/learn/tutorials/versioned/samza-sql.md
+++ b/docs/learn/tutorials/versioned/samza-sql.md
@@ -88,7 +88,7 @@ Before you can run a Samza application, you need to build a
package for it. Plea
After you've built your Samza package, you can start the app on the grid using
the run-app.sh script.
```bash
-./deploy/samza/bin/run-app.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/page-view-filter-sql.properties
+./deploy/samza/bin/run-app.sh
--config-path=$PWD/deploy/samza/config/page-view-filter-sql.properties
```
The app executes the following SQL command :
@@ -115,7 +115,7 @@ Congratulations! You've now setup a local grid that
includes YARN, Kafka, and Zo
To shutdown the app, use the same _run-app.sh_ script with an extra
_--operation=kill_ argument
```bash
-./deploy/samza/bin/run-app.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/page-view-filter-sql.properties
--operation=kill
+./deploy/samza/bin/run-app.sh
--config-path=$PWD/deploy/samza/config/page-view-filter-sql.properties
--operation=kill
```
Please follow the instructions from [Hello Samza High Level API - YARN
Deployment](hello-samza-high-level-yarn.html) on how to shutdown and cleanup
the app.
diff --git a/docs/startup/hello-samza/versioned/index.md
b/docs/startup/hello-samza/versioned/index.md
index 3573e26..4b986ca 100644
--- a/docs/startup/hello-samza/versioned/index.md
+++ b/docs/startup/hello-samza/versioned/index.md
@@ -52,7 +52,7 @@ Before you can run a Samza job, you need to build a package
for it. This package
**(Optional)** NOTE: if you want the hello-samza jobs to run with a local
Samza build (e.g., if you are a Samza developer),
make sure that you run the following steps, otherwise skip them.
{% highlight bash %}
-In your hello-world project,
+In your hello-samza project,
git checkout latest
In your local Samza project,
./gradlew publishToMavenLocal
@@ -63,7 +63,7 @@ Then, you can continue w/ the following command in
hello-samza project:
{% highlight bash %}
mvn clean package
mkdir -p deploy/samza
-tar -xvf ./target/hello-samza-1.4.0-SNAPSHOT-dist.tar.gz -C deploy/samza
+tar -xvf ./target/hello-samza-1.5.0-SNAPSHOT-dist.tar.gz -C deploy/samza
{% endhighlight %}
### Run a Samza Job
@@ -71,7 +71,7 @@ tar -xvf ./target/hello-samza-1.4.0-SNAPSHOT-dist.tar.gz -C
deploy/samza
After you've built your Samza package, you can start a job on the grid using
the run-app.sh script.
{% highlight bash %}
-deploy/samza/bin/run-app.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
+deploy/samza/bin/run-app.sh
--config-path=$PWD/deploy/samza/config/wikipedia-feed.properties
{% endhighlight %}
The job will consume a feed of real-time edits from Wikipedia, and produce
them to a Kafka topic called "wikipedia-raw". Give the job a minute to startup,
and then tail the Kafka topic:
@@ -89,8 +89,8 @@ If you can not see any output from Kafka consumer, you may
have connection probl
Let's calculate some statistics based on the messages in the wikipedia-raw
topic. Start two more jobs:
{% highlight bash %}
-deploy/samza/bin/run-app.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/wikipedia-parser.properties
-deploy/samza/bin/run-app.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/wikipedia-stats.properties
+deploy/samza/bin/run-app.sh
--config-path=$PWD/deploy/samza/config/wikipedia-parser.properties
+deploy/samza/bin/run-app.sh
--config-path=$PWD/deploy/samza/config/wikipedia-stats.properties
{% endhighlight %}
The first job (wikipedia-parser) parses the messages in wikipedia-raw, and
extracts information about the size of the edit, who made the change, etc. You
can take a look at its output with:
@@ -120,7 +120,7 @@ If you check the YARN UI, again, you'll see that all three
jobs are now listed.
To shutdown one of the jobs, use the same script with an extra
'--operation=kill' argument
{% highlight bash %}
-deploy/samza/bin/run-app.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
--operation=kill
+deploy/samza/bin/run-app.sh
--config-path=$PWD/deploy/samza/config/wikipedia-feed.properties
--operation=kill
{% endhighlight %}
After you're done, you can clean everything up using the same grid script.
diff --git a/docs/startup/quick-start/versioned/samza-sql.md
b/docs/startup/quick-start/versioned/samza-sql.md
index b3bfdbd..4517698 100644
--- a/docs/startup/quick-start/versioned/samza-sql.md
+++ b/docs/startup/quick-start/versioned/samza-sql.md
@@ -76,7 +76,7 @@ configuration file and submit it to a YARN cluster.
```bash
-./deploy/samza/bin/run-app.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/page-view-filter-sql.properties
+./deploy/samza/bin/run-app.sh
--config-path=$PWD/deploy/samza/config/page-view-filter-sql.properties
```
diff --git a/docs/startup/quick-start/versioned/samza.md
b/docs/startup/quick-start/versioned/samza.md
index 4c3c32a..394f238 100644
--- a/docs/startup/quick-start/versioned/samza.md
+++ b/docs/startup/quick-start/versioned/samza.md
@@ -211,7 +211,7 @@ Let’s kick off our application and use gradle to run it.
Alternately, you can
{% highlight bash %}
> export BASE_DIR=`pwd`
-> ./gradlew run
--args="--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$BASE_DIR/src/main/config/word-count.properties"
+> ./gradlew run --args="--config
job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory
--config
job.config.loader.properties.path=$BASE_DIR/src/main/config/word-count.properties"
{% endhighlight %}
diff --git
a/samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamWriter.java
b/samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamWriter.java
index 3d837ac..da65981 100644
---
a/samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamWriter.java
+++
b/samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamWriter.java
@@ -103,11 +103,9 @@ public class CoordinatorStreamWriter {
* Main function for using the CoordinatorStreamWriter. The main function
starts a CoordinatorStreamWriter
* and sends control messages.
* To run the code use the following command:
- * {path to samza deployment}/samza/bin/run-coordinator-stream-writer.sh
--config job.config.loader.factory={config--loader-factory} --config
job.config.loader.properties{properties needed for config loader to load
config} --type={type of the message} --key={[optional] key of the message}
--value={[optional] value of the message}
+ * {path to samza deployment}/samza/bin/run-coordinator-stream-writer.sh
--config job.config.loader.factory={config-loader-factory} --config
job.config.loader.properties{properties needed for config loader to load
config} --type={type of the message} --key={[optional] key of the message}
--value={[optional] value of the message}
*
* @param args input arguments for running the writer. These arguments are:
- * "config-factory" = The config file factory
- * "config-path" = The path to config file of a job
* "type" = type of the message being written
* "key" = [optional] key of the message being written
* "value" = [optional] value of the message being written
diff --git a/samza-rest/src/main/bash/run-samza-rest-service.sh
b/samza-rest/src/main/bash/run-samza-rest-service.sh
index bd52afd..a48daf3 100755
--- a/samza-rest/src/main/bash/run-samza-rest-service.sh
+++ b/samza-rest/src/main/bash/run-samza-rest-service.sh
@@ -16,7 +16,7 @@
# specific language governing permissions and limitations
# under the License.
-[[ $JAVA_OPTS != *-Dlog4j.configuration* ]] && export JAVA_OPTS="$JAVA_OPTS
-Dlog4j.configuration=file:$(dirname $0)/log4j.xml"
+[[ $JAVA_OPTS != *-Dlog4j.configuration* ]] && export JAVA_OPTS="$JAVA_OPTS
-Dlog4j.configuration=file:$(dirname $0)/log4j2.xml"
[[ -z "$SAMZA_LOG_DIR" ]] && export SAMZA_LOG_DIR="$PWD/logs"
exec $(dirname $0)/run-class.sh org.apache.samza.rest.SamzaRestService "$@"
diff --git
a/samza-rest/src/main/java/org/apache/samza/rest/SamzaRestService.java
b/samza-rest/src/main/java/org/apache/samza/rest/SamzaRestService.java
index a2db84b..b5a75d7 100644
--- a/samza-rest/src/main/java/org/apache/samza/rest/SamzaRestService.java
+++ b/samza-rest/src/main/java/org/apache/samza/rest/SamzaRestService.java
@@ -82,7 +82,8 @@ public class SamzaRestService {
* Command line interface to run the server.
*
* @param args arguments supported by {@link
org.apache.samza.util.CommandLine}.
- * In particular, --config-path and --config-factory are used to
read the Samza REST config file.
+ * In particular, --config job.config.loader.properties.path and
+ * --config job.config.loader.factory are used to read the Samza
REST config file.
* @throws Exception if the server could not be successfully started.
*/
public static void main(String[] args)
diff --git
a/samza-rest/src/main/java/org/apache/samza/rest/proxy/job/SimpleYarnJobProxy.java
b/samza-rest/src/main/java/org/apache/samza/rest/proxy/job/SimpleYarnJobProxy.java
index 310c3e6..e982d0f 100644
---
a/samza-rest/src/main/java/org/apache/samza/rest/proxy/job/SimpleYarnJobProxy.java
+++
b/samza-rest/src/main/java/org/apache/samza/rest/proxy/job/SimpleYarnJobProxy.java
@@ -37,11 +37,10 @@ import org.slf4j.LoggerFactory;
public class SimpleYarnJobProxy extends ScriptJobProxy {
private static final Logger log =
LoggerFactory.getLogger(SimpleYarnJobProxy.class);
- private static final String START_SCRIPT_NAME = "run-job.sh";
+ private static final String START_SCRIPT_NAME = "run-app.sh";
private static final String STOP_SCRIPT_NAME = "kill-yarn-job-by-name.sh";
- private static final String CONFIG_FACTORY_PARAM =
"--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory";
- private static final String CONFIG_PATH_PARAM_FORMAT =
"--config-path=file://%s";
+ private static final String CONFIG_PATH_PARAM_FORMAT = "--config-path=%s";
private final JobStatusProvider statusProvider;
@@ -64,7 +63,7 @@ public class SimpleYarnJobProxy extends ScriptJobProxy {
}
String scriptPath = getScriptPath(jobInstance, START_SCRIPT_NAME);
- int resultCode = scriptRunner.runScript(scriptPath, CONFIG_FACTORY_PARAM,
+ int resultCode = scriptRunner.runScript(scriptPath,
generateConfigPathParameter(jobInstance));
if (resultCode != 0) {
throw new SamzaException("Failed to start job. Result code: " +
resultCode);
diff --git a/samza-test/src/main/python/configs/tests.json
b/samza-test/src/main/python/configs/tests.json
index 5e89b97..2a9e52f 100644
--- a/samza-test/src/main/python/configs/tests.json
+++ b/samza-test/src/main/python/configs/tests.json
@@ -1,5 +1,5 @@
{
"samza_executable": "samza-test_2.11-1.5.0.tgz",
"samza_install_path": "deploy/smoke_tests",
- "samza_config_factory":
"org.apache.samza.config.factories.PropertiesConfigFactory"
+ "samza_config_loader_factory":
"org.apache.samza.config.loaders.PropertiesConfigLoaderFactory"
}
diff --git a/samza-test/src/main/python/deployment.py
b/samza-test/src/main/python/deployment.py
index 7cd3cac..0b72cc6 100644
--- a/samza-test/src/main/python/deployment.py
+++ b/samza-test/src/main/python/deployment.py
@@ -76,7 +76,7 @@ def setup_suite():
# Setup Samza job deployer.
samza_job_deployer = SamzaJobYarnDeployer({
- 'config_factory': c('samza_config_factory'),
+ 'config_loader_factory': c('samza_config_loader_factory'),
'yarn_site_template': c('yarn_site_template'),
'yarn_driver_configs': c('yarn_driver_configs'),
'yarn_nm_hosts': c('yarn_nm_hosts').values(),
diff --git a/samza-test/src/main/python/samza_job_yarn_deployer.py
b/samza-test/src/main/python/samza_job_yarn_deployer.py
index 405678f..8a785ac 100644
--- a/samza-test/src/main/python/samza_job_yarn_deployer.py
+++ b/samza-test/src/main/python/samza_job_yarn_deployer.py
@@ -35,11 +35,11 @@ logger = logging.getLogger(__name__)
class SamzaJobYarnDeployer(Deployer):
def __init__(self, configs={}):
"""
- Instantiates a Samza job deployer that uses run-job.sh and
kill-yarn-job.sh
+ Instantiates a Samza job deployer that uses run-job.sh and kill-yarn-job.sh
to start and stop Samza jobs in a YARN grid.
param: configs -- Map of config key/values pairs. These configs will be
used
- as a default whenever overrides are not provided in the methods (install,
+ as a default whenever overrides are not provided in the methods (install,
start, stop, etc) below.
"""
logging.getLogger("paramiko").setLevel(logging.ERROR)
@@ -52,7 +52,7 @@ class SamzaJobYarnDeployer(Deployer):
def install(self, package_id, configs={}):
"""
- Installs a package (tarball, or zip) on to a list of remote hosts by
+ Installs a package (tarball, or zip) on to a list of remote hosts by
SFTP'ing the package to the remote install_path.
param: package_id -- A unique ID used to identify an installed YARN
package.
@@ -104,29 +104,29 @@ class SamzaJobYarnDeployer(Deployer):
with a package_id, and a config file.
param: configs -- Map of config key/values pairs. Valid keys include:
- package_id: The package_id for the package that contains the code for
job_id.
- Usually, the package_id refers to the .tgz job tarball that contains the
+ package_id: The package_id for the package that contains the code for
job_id.
+ Usually, the package_id refers to the .tgz job tarball that contains the
code necessary to run job_id.
- config_factory: The config factory to use to decode the config_file.
+ config_loader_factory: The config loader factory to use to decode the
config_file.
config_file: Path to the config file for the job to be run.
install_path: Path where the package for the job has been installed on
remote NMs.
- properties: (optional) [(property-name,property-value)] Optional override
- properties for the run-job.sh script. These properties override the
+ properties: (optional) [(property-name,property-value)] Optional override
+ properties for the run-job.sh script. These properties override the
config_file's properties.
"""
configs = self._get_merged_configs(configs)
- self._validate_configs(configs, ['package_id', 'config_factory',
'config_file', 'install_path'])
+ self._validate_configs(configs, ['package_id', 'config_loader_factory',
'config_file', 'install_path'])
# Get configs.
package_id = configs.get('package_id')
- config_factory = configs.get('config_factory')
+ config_loader_factory = configs.get('config_loader_factory')
config_file = configs.get('config_file')
install_path = configs.get('install_path')
properties = configs.get('properties', {})
properties['yarn.package.path'] = 'file:' + os.path.join(install_path,
self._get_package_tgz_name(package_id))
# Execute bin/run-job.sh locally from driver machine.
- command = "{0} --config-factory={1}
--config-path={2}".format(os.path.join(package_id, "bin/run-job.sh"),
config_factory, os.path.join(package_id, config_file))
+ command = "{0} --config job.config.loader.factory={1} --config
job.config.loader.properties.path={2}".format(os.path.join(package_id,
"bin/run-job.sh"), config_loader_factory, os.path.join(package_id, config_file))
env = self._get_env_vars(package_id)
for property_name, property_value in properties.iteritems():
command += " --config {0}={1}".format(property_name, property_value)
@@ -176,7 +176,7 @@ class SamzaJobYarnDeployer(Deployer):
def await(self, job_id, configs={}):
"""
- Waits for a Samza job to finish using bin/stat-yarn-job.sh. A job is
+ Waits for a Samza job to finish using bin/stat-yarn-job.sh. A job is
finished when its "Final State" is not "UNDEFINED".
param: job_id -- A unique ID used to idenitfy a Samza job.
diff --git
a/samza-yarn/src/main/java/org/apache/samza/validation/YarnJobValidationTool.java
b/samza-yarn/src/main/java/org/apache/samza/validation/YarnJobValidationTool.java
index bbcc976..23f0e4d 100644
---
a/samza-yarn/src/main/java/org/apache/samza/validation/YarnJobValidationTool.java
+++
b/samza-yarn/src/main/java/org/apache/samza/validation/YarnJobValidationTool.java
@@ -61,7 +61,7 @@ import org.slf4j.LoggerFactory;
*
* When running this tool, please provide the configuration URI of job. For
example:
*
- * deploy/samza/bin/validate-yarn-job.sh
--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
--config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
[--metrics-validator=com.foo.bar.SomeMetricsValidator]
+ * deploy/samza/bin/validate-yarn-job.sh --config
job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory
--config
job.config.loader.properties.path=$PWD/deploy/samza/config/wikipedia-feed.properties
[--metrics-validator=com.foo.bar.SomeMetricsValidator]
*
* The tool prints out the validation result in each step and throws an
exception when the
* validation fails.
