Hi, Problem resolved. The root cause: there is old data in zookeeper before flume connect zookeeper. the work around here is remove old data file in zookeeper and restart flume/zookeeper.
At 2012-05-10 16:30:10,"alo alt" <[email protected]> wrote: >If you use the external zookeeper you have to configure them. If you use the >internal, flume will do that for you. >Setup and configure zookeeper: >http://zookeeper.apache.org/doc/r3.1.2/zookeeperStarted.html > >Another good one: >http://sanjivblogs.blogspot.de/2011/04/quick-flume-installation-and.html > >-- >Alexander Lorenz >http://mapredit.blogspot.com >German Hadoop LinkedIn Group: http://goo.gl/N8pCF > >On May 10, 2012, at 10:07 AM, lulynn_2008 wrote: > >> Hi, >> I am running flume in Virtual Machines with DHCP network. >> -- cat /etc/sysconfig/network-scripts/ifcfg-eth0 >> # Intel Corporation 82575EB Gigabit Network Connection >> DEVICE=eth0 >> BOOTPROTO=dhcp >> DHCPCLASS= >> HWADDR=5C:F3:FC:A6:5D:C0 >> ONBOOT=yes >> -- The system information: >> Linux version 2.6.18-194.el5 ([email protected]) (gcc >> version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Tue Mar 16 21:52:39 EDT >> 2010 >> >> Following is all the configuration about zookeeper in flume-conf.xml: >> <property> >> <name>flume.master.store</name> >> <value>zookeeper</value> >> <description>How the Flume Master stores node configurations. Must >> be either 'zookeeper' or 'memory'.</description> >> </property> >> >> <property> >> <name>flume.master.zk. >> logdir</name> >> >> <value>/flume/flume-${user.name}-zk</value> >> <description>The base directory in which the ZBCS stores >> data.</description> >> </property> >> >> <property> >> <name>flume.master.zk.use.external</name> >> <value>true</value> >> </property> >> >> >> <property> >> <name>flume.master.zk.servers</name> >> <value>node1:2181,node2:2181,node3:2181</value> >> </property> >> >> >> >> >> At 2012-05-10 13:58:40,"alo alt" <[email protected]> wrote: >> >Add your zookeeper config too. Do you run them in a VM or bare metal? DHCP >> >network? >> > >> > >> >-- >> >Alexander Lorenz >> >http://mapredit.blogspot.com >> >German Hadoop LinkedIn Group: http://goo.gl/N8pCF >> > >> >On May 10, 2012, at 3:46 AM, lulynn_2008 wrote: >> > >> >> Yes, sure. >> >> >> >> >> >> >> >> At 2012-05-09 18:40:31,"alo alt" <[email protected]> wrote: >> >> >Did you add >> >> > >> >> ><property> >> >> ><name>flume.master.zk.use.external</name> >> >> ><value>true</value> >> >> ></property> >> >> > >> >> >into the flume master config? >> >> > >> >> >-- >> >> >Alexander Lorenz >> >> >http://mapredit.blogspot.com >> >> >German Hadoop LinkedIn Group: http://goo.gl/N8pCF >> >> > >> >> >On May 9, 2012, at 12:22 PM, lulynn_2008 wrote: >> >> > >> >> >> Hi, >> >> >> After remove ";" in < ;/configuration>, and restart flume based on >> >> >> https://cwiki.apache.org/FLUME/troubleshooting-faq.html#TroubleshootingFAQ-Ihaveencountereda%2522Couldnotincrementversioncounter%2522errormessage. >> >> >> There is still the same error during starting flume master. >> >> >> >> >> >> >> >> >> >> >> >> At 2012-05-09 16:39:03,"alo alt" <[email protected]> wrote: >> >> >> >Hi, >> >> >> > >> >> >> >Please remove the ";" in >> >> >> >> < ;/configuration> >> >> >> > >> >> >> >That could be a zookeeper issue on virtual machines: >> >> >> >https://cwiki.apache.org/FLUME/troubleshooting-faq.html#TroubleshootingFAQ-Ihaveencountereda%2522Couldnotincrementversioncounter%2522errormessage. >> >> >> > >> >> >> >- Alex >> >> >> > >> >> >> >-- >> >> >> >Alexander Lorenz >> >> >> >http://mapredit.blogspot.com >> >> >> >German Hadoop LinkedIn Group: http://goo.gl/N8pCF >> >> >> > >> >> >> >On May 9, 2012, at 10:31 AM, lulynn_2008 wrote: >> >> >> > >> >> >> >> Hi All, >> >> >> >> Please help to check why this happened. thank you. >> >> >> >> -- I am using external zookeeper to run flume. During start >> >> >> >> flume master node, I got following error information in >> >> >> >> flumemaster.out. And we can find flume master has been started by >> >> >> >> pid files. >> >> >> >> -- But when run "refreshAll" in flume shell, the command failed >> >> >> >> with "Could not increment version counter...". >> >> >> >> -- When run flume without external zookeeper, flume can work >> >> >> >> normally. So I assume this is caused by zookeeper part. >> >> >> >> Here is the zookeeper configuration: node1 is hostname and >> >> >> >> node2/node3 is ip. >> >> >> >> <property> >> >> >> >> <name>flume.master.zk.servers</name> >> >> >> >> <value>node1:2181,node2:2181,node3:2181</value> >> >> >> >> </property> >> >> >> >> < ;/configuration> >> >> >> >> >> >> >> >> ERROR INFORMATION: >> >> >> >> 2012-05-09 01:09:39,418 [main] ERROR master.ConfigManager: >> >> >> >> ConfigStore init threw IOException >> >> >> >> java.io.IOException: Unexpected exception in loadConfigs >> >> >> >> at >> >> >> >> com.cloudera.flume.master.ZooKeeperConfigStore.loadConfigs(ZooKeeperConfigStore.java:267) >> >> >> >> at >> >> >> >> com.cloudera.flume.master.ZooKeeperConfigStore$1.success(ZooKeeperConfigStore.java:138) >> >> >> >> at >> >> >> >> com.cloudera.flume.master.ZKClient$1.doTry(ZKClient.java:169) >> >> >> >> at >> >> >> >> com.cloudera.util.RetryHarness.attempt(RetryHarness.java:64) >> >> >> >> at >> >> >> >> com.cloudera.flume.master.ZKClient.init(ZKClient.java:178) >> >> >> >> at >> >> >> >> com.cloudera.flume.master.ZooKeeperConfigStore.connect(ZooKeeperConfigStore.java:145) >> >> >> >> at com.c >> >> >> >> loudera.flume.master.ZooKeeperConfigStore.init(ZooKeeperConfigStore.java:110) >> >> >> >> at >> >> >> >> com.cloudera.flume.master.ConfigManager.start(ConfigManager.java:375) >> >> >> >> at >> >> >> >> com.cloudera.flume.master.flows.FlowConfigManager.start(FlowConfigManager.java:291) >> >> >> >> at >> >> >> >> com.cloudera.flume.master.TranslatingConfigurationManager.start(TranslatingConfigurationManager.java:391) >> >> >> >> at >> >> >> >> com.cloudera.flume.master.FlumeMaster.serve(FlumeMaster.java:295) >> >> >> >> at >> >> >> >> com.cloudera.flume.master.FlumeMaster.main(FlumeMaster.java:556) >> >> >> >> Caused by: java.io.IOException: Not a data file. >> >> >> >> at >> >> >> >> org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:105) >> >> >> >> at org.apache.avro.file.DataFileStream.<init>(DataFileStre >> >> >> >> am.java:84) >> >> >> >> at >> >> >> >> com.cloudera.flume.master.ZooKeeperConfigStore.deserializeConfigs(ZooKeeperConfigStore.java:190) >> >> >> >> at >> >> >> >> com.cloudera.flume.master.ZooKeeperConfigStore.loadConfigs(ZooKeeperConfigStore.java:265) >> >> >> >> ... 11 more >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> > >> >> >> >
