If you use the external zookeeper you have to configure them. If you use the internal, flume will do that for you. Setup and configure zookeeper: http://zookeeper.apache.org/doc/r3.1.2/zookeeperStarted.html
Another good one: http://sanjivblogs.blogspot.de/2011/04/quick-flume-installation-and.html -- Alexander Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF On May 10, 2012, at 10:07 AM, lulynn_2008 wrote: > Hi, > I am running flume in Virtual Machines with DHCP network. > -- cat /etc/sysconfig/network-scripts/ifcfg-eth0 > # Intel Corporation 82575EB Gigabit Network Connection > DEVICE=eth0 > BOOTPROTO=dhcp > DHCPCLASS= > HWADDR=5C:F3:FC:A6:5D:C0 > ONBOOT=yes > -- The system information: > Linux version 2.6.18-194.el5 ([email protected]) (gcc > version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Tue Mar 16 21:52:39 EDT 2010 > > Following is all the configuration about zookeeper in flume-conf.xml: > <property> > <name>flume.master.store</name> > <value>zookeeper</value> > <description>How the Flume Master stores node configurations. Must > be either 'zookeeper' or 'memory'.</description> > </property> > > <property> > <name>flume.master.zk. > logdir</name> > > <value>/flume/flume-${user.name}-zk</value> > <description>The base directory in which the ZBCS stores > data.</description> > </property> > > <property> > <name>flume.master.zk.use.external</name> > <value>true</value> > </property> > > > <property> > <name>flume.master.zk.servers</name> > <value>node1:2181,node2:2181,node3:2181</value> > </property> > > > > > At 2012-05-10 13:58:40,"alo alt" <[email protected]> wrote: > >Add your zookeeper config too. Do you run them in a VM or bare metal? DHCP > >network? > > > > > >-- > >Alexander Lorenz > >http://mapredit.blogspot.com > >German Hadoop LinkedIn Group: http://goo.gl/N8pCF > > > >On May 10, 2012, at 3:46 AM, lulynn_2008 wrote: > > > >> Yes, sure. > >> > >> > >> > >> At 2012-05-09 18:40:31,"alo alt" <[email protected]> wrote: > >> >Did you add > >> > > >> ><property> > >> ><name>flume.master.zk.use.external</name> > >> ><value>true</value> > >> ></property> > >> > > >> >into the flume master config? > >> > > >> >-- > >> >Alexander Lorenz > >> >http://mapredit.blogspot.com > >> >German Hadoop LinkedIn Group: http://goo.gl/N8pCF > >> > > >> >On May 9, 2012, at 12:22 PM, lulynn_2008 wrote: > >> > > >> >> Hi, > >> >> After remove ";" in < ;/configuration>, and restart flume based on > >> >> https://cwiki.apache.org/FLUME/troubleshooting-faq.html#TroubleshootingFAQ-Ihaveencountereda%2522Couldnotincrementversioncounter%2522errormessage. > >> >> There is still the same error during starting flume master. > >> >> > >> >> > >> >> > >> >> At 2012-05-09 16:39:03,"alo alt" <[email protected]> wrote: > >> >> >Hi, > >> >> > > >> >> >Please remove the ";" in > >> >> >> < ;/configuration> > >> >> > > >> >> >That could be a zookeeper issue on virtual machines: > >> >> >https://cwiki.apache.org/FLUME/troubleshooting-faq.html#TroubleshootingFAQ-Ihaveencountereda%2522Couldnotincrementversioncounter%2522errormessage. > >> >> > > >> >> >- Alex > >> >> > > >> >> >-- > >> >> >Alexander Lorenz > >> >> >http://mapredit.blogspot.com > >> >> >German Hadoop LinkedIn Group: http://goo.gl/N8pCF > >> >> > > >> >> >On May 9, 2012, at 10:31 AM, lulynn_2008 wrote: > >> >> > > >> >> >> Hi All, > >> >> >> Please help to check why this happened. thank you. > >> >> >> -- I am using external zookeeper to run flume. During start flume > >> >> >> master node, I got following error information in flumemaster.out. > >> >> >> And we can find flume master has been started by pid files. > >> >> >> -- But when run "refreshAll" in flume shell, the command failed > >> >> >> with "Could not increment version counter...". > >> >> >> -- When run flume without external zookeeper, flume can work > >> >> >> normally. So I assume this is caused by zookeeper part. > >> >> >> Here is the zookeeper configuration: node1 is hostname and > >> >> >> node2/node3 is ip. > >> >> >> <property> > >> >> >> <name>flume.master.zk.servers</name> > >> >> >> <value>node1:2181,node2:2181,node3:2181</value> > >> >> >> </property> > >> >> >> < ;/configuration> > >> >> >> > >> >> >> ERROR INFORMATION: > >> >> >> 2012-05-09 01:09:39,418 [main] ERROR master.ConfigManager: > >> >> >> ConfigStore init threw IOException > >> >> >> java.io.IOException: Unexpected exception in loadConfigs > >> >> >> at > >> >> >> com.cloudera.flume.master.ZooKeeperConfigStore.loadConfigs(ZooKeeperConfigStore.java:267) > >> >> >> at > >> >> >> com.cloudera.flume.master.ZooKeeperConfigStore$1.success(ZooKeeperConfigStore.java:138) > >> >> >> at > >> >> >> com.cloudera.flume.master.ZKClient$1.doTry(ZKClient.java:169) > >> >> >> at > >> >> >> com.cloudera.util.RetryHarness.attempt(RetryHarness.java:64) > >> >> >> at com.cloudera.flume.master.ZKClient.init(ZKClient.java:178) > >> >> >> at > >> >> >> com.cloudera.flume.master.ZooKeeperConfigStore.connect(ZooKeeperConfigStore.java:145) > >> >> >> at com.c > >> >> >> loudera.flume.master.ZooKeeperConfigStore.init(ZooKeeperConfigStore.java:110) > >> >> >> at > >> >> >> com.cloudera.flume.master.ConfigManager.start(ConfigManager.java:375) > >> >> >> at > >> >> >> com.cloudera.flume.master.flows.FlowConfigManager.start(FlowConfigManager.java:291) > >> >> >> at > >> >> >> com.cloudera.flume.master.TranslatingConfigurationManager.start(TranslatingConfigurationManager.java:391) > >> >> >> at > >> >> >> com.cloudera.flume.master.FlumeMaster.serve(FlumeMaster.java:295) > >> >> >> at > >> >> >> com.cloudera.flume.master.FlumeMaster.main(FlumeMaster.java:556) > >> >> >> Caused by: java.io.IOException: Not a data file. > >> >> >> at > >> >> >> org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:105) > >> >> >> at org.apache.avro.file.DataFileStream.<init>(DataFileStre > >> >> >> am.java:84) > >> >> >> at > >> >> >> com.cloudera.flume.master.ZooKeeperConfigStore.deserializeConfigs(ZooKeeperConfigStore.java:190) > >> >> >> at > >> >> >> com.cloudera.flume.master.ZooKeeperConfigStore.loadConfigs(ZooKeeperConfigStore.java:265) > >> >> >> ... 11 more > >> >> >> > >> >> >> > >> >> >> > >> >> > > >> >> > >> >> > >> >> > >> > > >> > >> > >> > > > > >
