On Fri, Jan 23, 2015 at 2:42 PM, Sa Li <sal...@gmail.com> wrote:
> Also I found ./kafka/system_test/cluster_config.json is duplicated on each
> directory ./kafka/system_test/replication_testsuite/testcase_xxxx/
>
Not duplicated, but customized:
$ diff -u system_test/cluster_config.json
system_test/replication_testsuite//testcase_0021/cluster_config.json
--- system_test/cluster_config.json 2014-12-17 17:43:01.000000000 -0800
+++ system_test/replication_testsuite//testcase_0021/cluster_config.json
2014-12-17 17:43:01.000000000 -0800
@@ -48,11 +48,29 @@
{
"entity_id": "5",
"hostname": "localhost",
- "role": "console_consumer",
+ "role": "producer_performance",
"cluster_name": "source",
"kafka_home": "default",
"java_home": "default",
"jmx_port": "9998"
+ },
+ {
+ "entity_id": "6",
+ "hostname": "localhost",
+ "role": "console_consumer",
+ "cluster_name": "source",
+ "kafka_home": "default",
+ "java_home": "default",
+ "jmx_port": "9999"
+ },
+ {
+ "entity_id": "7",
+ "hostname": "localhost",
+ "role": "console_consumer",
+ "cluster_name": "source",
+ "kafka_home": "default",
+ "java_home": "default",
+ "jmx_port": "9099"
}
]
}
> When I change the ./kafka/system_test/cluster_config.json, do I need to
> overwrite it each
> ./kafka/system_test/replication_testsuite/testcase_xxxx/cluster_config.json
> ?
You'll need to edit all the cluster_config.json files for directories of
the tests you want to run. Note that this applies to entire test suites
too, e.g. there is
system_test/offset_management_testsuite/cluster_config.json which overrides
the the default cluster_config.json for all tests in
offset_management_testsuite.
> Thanks
>
> AL
>
>
>
> On Fri, Jan 23, 2015 at 1:39 PM, Sa Li <sal...@gmail.com> wrote:
>
> > Thanks for reply. Ewen, pertaining to your statement "... hostname
> setting
> > being a list instead of a single host," are you saying entity_id 1 or 0,
> >
> > "entity_id": "0",
> > "hostname":
> > "10.100.70.28,10.100.70.29,10.100.70.30,10.100.70.31,10.100.70.32",
> >
> > "entity_id": "1",
> > "hostname": "10.100.70.28",
>
"Entity" here means a single service on a single machine, i.e 1 zookeeper
instance. That's why the default cluster_config.json has 3 separate broker
entries.
-Ewen
> >
> > I thought the role zookeeper has multiple hosts, so I list all the IPs of
> > ensemble. While entity 1 is only about 1 broker (my design about
> production
> > cluster to fire up one broker for each host, so 3 nodes with 3 brokers),
> so
> > I specify one hostname IP only here. How do I change?
> >
> >
> > thanks
> >
> > AL
> >
> > On Fri, Jan 23, 2015 at 1:22 PM, Ewen Cheslack-Postava <
> e...@confluent.io>
> > wrote:
> >
> >> 1. Except for that hostname setting being a list instead of a single
> host,
> >> the changes look reasonable. That is where you want to customize
> settings
> >> for your setup.
> >>
> >> 2 & 3. Yes, you'll want to update those files as well. They top-level
> ones
> >> provide defaults, the ones in specific test directories provide
> overrides
> >> for that specific test. But they aren't combined in any way, i.e. the
> more
> >> specific one is just taken as a whole rather than being like a diff, so
> >> you
> >> do have to update both.
> >>
> >> You might want to take a look at
> >> https://issues.apache.org/jira/browse/KAFKA-1748. Currently if you want
> >> to
> >> run all tests it's a pain to change the hosts they're running on since
> it
> >> requires manually editing all those files. The patch gets rid of
> >> cluster_config.json and provides a couple of different ways of
> configuring
> >> the cluster -- run everything on localhost, get cluster info from a
> single
> >> json file, or get the ssh info from Vagrant.
> >>
> >>
> >>
> >> On Fri, Jan 23, 2015 at 11:50 AM, Sa Li <sal...@gmail.com> wrote:
> >>
> >> > Hi, All
> >> >
> >> > From my last ticket (Subject: kafka production server test), Guozhang
> >> > kindly point me the system test package come with kafka source build
> >> which
> >> > is really cool package. I took a look at this package, things are
> clear
> >> is
> >> > I run it on localhost, I don't need to change anything, say,
> >> > cluster_config.json defines entities, and system test reads
> >> > testcase_xxxx_properties.json to override the properties in
> >> > cluster_config.json. For example, cluster_config.json defaults
> hostname
> >> as
> >> > localhost, and three brokers, I assume it will create 3 brokers in
> >> > localhost and continue the test.
> >> >
> >> > Currently I install the package on a vagrant VM, and like to run the
> >> system
> >> > test on VM and remotely access production to test production cluster.
> >> The
> >> > production cluster has 3 nodes. kafka production cluster is on top of
> a
> >> > 5-node zookeeper ensemble. My questions is how to effectively change
> >> the
> >> > properties on vagrant system test package.
> >> >
> >> > 1. change on cluster_config.json, like
> >> > {
> >> > "entity_id": "0",
> >> > "hostname":
> >> > "10.100.70.28,10.100.70.29,10.100.70.30,10.100.70.31,10.100.70.32",
> >> > "role": "zookeeper",
> >> > "cluster_name": "target",
> >> > "kafka_home": "/etc/kafka",
> >> > "java_home": "/usr/lib/jvm/java-7-openjdk-amd64/jre",
> >> > "jmx_port": "9990"
> >> > },
> >> > {
> >> > "entity_id": "1",
> >> > "hostname": "10.100.70.28",
> >> > "role": "broker",
> >> > "cluster_name": "target",
> >> > "kafka_home": "/etc/kafka",
> >> > "java_home": "/usr/lib/jvm/java-7-openjdk-amd64/jre",
> >> > "jmx_port": "9991"
> >> > },
> >> >
> >> > Here because I want to test remote servers, so I need to change the
> >> > cluster_name as "target", right?
> >> >
> >> > 2. In directory ./replication_testsuite/config/ , for all the
> >> properties
> >> > files, do I need to change them all to be the same as the properties
> on
> >> > production servers?
> >> >
> >> > 3. in ./replication_testsuite/testcase_xxxx/, seems I need to make
> >> > corresponding changes as well to keep consistent with
> >> > ./config/....properties, such as
> >> > "log.dir": "/tmp/kafka_server_1_logs" will be change to the log.dir in
> >> my
> >> > production server.properties, is that right?
> >> >
> >> >
> >> > Hope someone who has done the system test on remote server can share
> >> some
> >> > experience, thanks
> >> >
> >> >
> >> >
> >> > AL
> >> >
> >> > --
> >> >
> >> > Alec Li
> >> >
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Ewen
> >>
> >
> >
> >
> > --
> >
> > Alec Li
> >
>
>
>
> --
>
> Alec Li
>
--
Thanks,
Ewen
