I found that if I change windows agent configure
fromtail("c:\\test.txt") agentSink("hadoop48",35853)
;to
tail("c:\\test.txt")agentDFOSink("hadoop48",35853) ;
then the agent changes status from OPENING to ACTIVE.
my windows agent is :192.168.20.81, while linux collector is : 192.168.10.48
another linux agent is 192.168.10.46. 10.46 have no problem of agentSink.Thanks! Andy. 2013/1/25 周梦想 <[email protected]> > when i start the flume node service,the agent is OPENING. > Hello, > > I need to gather logs from windows 7, but the flume master and collector > is running on linux. I download 0.9.4 exe file from flume official web. > > When I start the node from windows, the status is OPENING. > > zhouhh.TKOffice.localzhouhh.TKOffice.local zhouhh.TKOffice.localOPENINGFri > Jan 25 11:45:01 CST 2013 4 > > And the log reports OutOfMemoryError. > > ================ > D:\Program Files\Cloudera\Flume 0.9.4\log\flumenode-stdout.2013-01-25.log > > 2013-01-25 15:31:03 Commons Daemon procrun stdout initializedListening for > transport dt_socket at address: 8888 > java.lang.OutOfMemoryError: Java heap space > Dumping heap to java_pid8100.hprof ... > Heap dump file created [13270531 bytes in 0.115 secs] > > =============== > D:\Program Files\Cloudera\Flume 0.9.4\log\flumenode-stderr.2013-01-25.log > > 2013-01-25 15:31:03 Commons Daemon procrun stderr initialized > 13/01/25 15:31:03 INFO agent.FlumeNode: Flume 0.9.4 > 13/01/25 15:31:03 INFO agent.FlumeNode: rev > 8d0aa4be4ee50bb664cb0d1624e5634f46d7b62a > 13/01/25 15:31:03 INFO agent.FlumeNode: Compiled on Mon Jun 20 14:46:53 > PDT 2011 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.runtime.name=Java(TM) > SE Runtime Environment > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.boot.library.path=C:\PROGRA~1\Java\jre7\bin > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.vm.version=23.3-b01 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.vm.vendor=Oracle Corporation > 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vendor.url= > http://java.oracle.com/ > 13/01/25 15:31:03 INFO agent.FlumeNode: System property path.separator=; > 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vm.name=Java > HotSpot(TM) Client VM > 13/01/25 15:31:03 INFO agent.FlumeNode: System property file.encoding.pkg= > sun.io > 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.country=CN > 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.script= > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.os.patch.level=Service Pack 1 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.vm.specification.name=Java Virtual Machine Specification > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > user.dir=C:\Windows\system32 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.runtime.version=1.7.0_07-b11 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property flume= > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.awt.graphicsenv=sun.awt.Win32GraphicsEnvironment > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.endorsed.dirs=C:\PROGRA~1\Java\jre7\lib\endorsed > 13/01/25 15:31:03 INFO agent.FlumeNode: System property os.arch=x86 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.io.tmpdir=C:\Windows\TEMP\ > 13/01/25 15:31:03 INFO agent.FlumeNode: System property line.separator= > > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.vm.specification.vendor=Oracle Corporation > 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.variant= > 13/01/25 15:31:03 INFO agent.FlumeNode: System property os.name=Windows 7 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > flume.home=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\.. > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.jnu.encoding=GBK > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.library.path=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program > Files\Common > Files\NetSarang;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program > Files\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL > Server\100\DTS\Binn\;C:\Program Files\Common Files\Thunder > Network\KanKan\Codecs;d:\php;d:\php\ext;D:\Program Files\MySQL\MySQL Server > 5.5\bin;D:\mysql\bin;D:\wxWidgets-2.9.1\include;D:\wxWidgets-2.9.1\include\msvc;W:\Server\include;W:\Base\include;W:\Client\include;d:\Program > Files\Lua\5.1;d:\Program Files\Lua\5.1\clibs;D:\Program > Files\TortoiseSVN\bin;d:\Program Files\Git\cmd;d:\Program > Files\Git\bin;D:\Program Files\TortoiseGit\bin;D:\docbook\bin;;. > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.specification.name=Java Platform API Specification > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.class.version=51.0 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.management.compiler=HotSpot Client Compiler > 13/01/25 15:31:03 INFO agent.FlumeNode: System property os.version=6.1 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.home=C:\ > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > user.timezone=Asia/Shanghai > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.awt.printerjob=sun.awt.windows.WPrinterJob > 13/01/25 15:31:03 INFO agent.FlumeNode: System property file.encoding=GBK > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.specification.version=1.7 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.class.path=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\build\classes;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\activation-1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\ahocorasick-2.x.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\ant-1.6.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-2.7.7.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-runtime-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\asm-3.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\automaton-1.0.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-compiler-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-ipc-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-cli-1.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-codec-1.3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-collections-3.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-el-1.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-httpclient-3.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-lang-2.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-logging-1.0.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-net-1.4.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\core-3.1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\flume-core-0.9.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\guava-r07.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\hadoop-core-0.20.2-CDH3B4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\hsqldb-1.8.0.7.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\irclib-1.0.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-core-asl-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-jaxrs-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-mapper-asl-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-xc-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jasper-compiler-5.5.23.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jasper-runtime-5.5.23.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jaxb-api-2.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jaxb-impl-2.1.12.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jersey-core-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jersey-json-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jersey-server-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jets3t-0.6.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jettison-1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jetty-6.1.26.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jetty-util-6.1.26.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jline-0.9.94.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jms-1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jsp-api-2.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jsp-api-2.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\libthrift-0.6.0-cdh.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\log4j-1.2.15.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\mail-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\netty-3.2.4.Final.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\oro-2.0.8.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\paranamer-2.3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\servlet-api-2.5-20081211.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\servlet-api-2.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\slf4j-api-1.5.11.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\slf4j-log4j12-1.5.11.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\snappy-java-1.0.1-rc3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\stax-api-1.0-2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\stax-api-1.0.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\stringtemplate-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\velocity-1.6.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\xmlenc-0.52.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\zookeeper-3.3.1-10.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\conf > 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.name=ZHOUHH$ > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > flume.root.logger=INFO,console > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.vm.specification.version=1.7 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.home=C:\PROGRA~1\Java\jre7 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.arch.data.model=32 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.language=zh > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.specification.vendor=Oracle Corporation > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > awt.toolkit=sun.awt.windows.WToolkit > 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vm.info=mixed > mode > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.version=1.7.0_07 > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.ext.dirs=C:\PROGRA~1\Java\jre7\lib\ext;C:\Windows\Sun\Java\lib\ext > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.boot.class.path=C:\PROGRA~1\Java\jre7\lib\alt-rt.jar;C:\PROGRA~1\Java\jre7\lib\resources.jar;C:\PROGRA~1\Java\jre7\lib\rt.jar;C:\PROGRA~1\Java\jre7\lib\sunrsasign.jar;C:\PROGRA~1\Java\jre7\lib\jsse.jar;C:\PROGRA~1\Java\jre7\lib\jce.jar;C:\PROGRA~1\Java\jre7\lib\charsets.jar;C:\PROGRA~1\Java\jre7\lib\jfr.jar;C:\PROGRA~1\Java\jre7\classes > 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vendor=Oracle > Corporation > 13/01/25 15:31:03 INFO agent.FlumeNode: System property file.separator=\ > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > java.vendor.url.bug=http://bugreport.sun.com/bugreport/ > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.io.unicode.encoding=UnicodeLittle > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.cpu.endian=little > 13/01/25 15:31:03 INFO agent.FlumeNode: System property sun.desktop=windows > 13/01/25 15:31:03 INFO agent.FlumeNode: System property > sun.cpu.isalist=pentium_pro+mmx pentium_pro pentium+mmx pentium i486 i386 > i86 > 13/01/25 15:31:03 INFO agent.FlumeNode: Starting flume agent on: > zhouhh.TKOffice.local > 13/01/25 15:31:03 INFO agent.FlumeNode: Working directory is: > C:\Windows\system32\. > 13/01/25 15:31:03 INFO conf.FlumeConfiguration: Loading configurations > from d:/PROGRA~1/Cloudera/FLUME0~1.4/conf > 13/01/25 15:31:03 WARN text.FormatFactory: Unable to load output format > plugin class - Class not found > 13/01/25 15:31:03 INFO mortbay.log: Logging to > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > org.mortbay.log.Slf4jLog > 13/01/25 15:31:03 INFO util.InternalHttpServer: Starting internal HTTP > server > 13/01/25 15:31:03 INFO mortbay.log: jetty-6.1.26 > 13/01/25 15:31:03 INFO mortbay.log: Extract > d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\webapps\flumeagent.war to > C:\Windows\TEMP\Jetty_0_0_0_0_35862_flumeagent.war__flumeagent__4cgrz3\webapp > 13/01/25 15:31:03 INFO mortbay.log: Started > [email protected]:35862 > 13/01/25 15:31:03 INFO util.InternalHttpServer: Server started > 13/01/25 15:31:03 INFO agent.MultiMasterRPC: No active master RPC > connection > 13/01/25 15:31:04 INFO agent.ThriftMasterRPC: Connected to master at > hadoop48:35872 > 13/01/25 15:31:04 INFO agent.LogicalNodeManager: creating new logical node > zhouhh.TKOffice.local > 13/01/25 15:31:04 INFO agent.FlumeNode: Hadoop Security enabled: false > 13/01/25 15:31:04 INFO util.InternalHttpServer: Starting internal HTTP > server > 13/01/25 15:31:04 INFO mortbay.log: jetty-6.1.26 > 13/01/25 15:31:04 INFO mortbay.log: Extract > d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\webapps\flumeagent.war to > C:\Windows\TEMP\Jetty_0_0_0_0_35862_flumeagent.war__flumeagent__4cgrz3\webapp > 13/01/25 15:31:04 INFO mortbay.log: Started > [email protected]:35862 > 13/01/25 15:31:04 INFO util.InternalHttpServer: Server started > 13/01/25 15:31:09 INFO agent.LogicalNode: Node config successfully set to > com.cloudera.flume.conf.FlumeConfigData@231214 > 13/01/25 15:31:09 INFO durability.NaiveFileWALManager: NaiveFileWALManager > is now open > 13/01/25 15:31:09 INFO durability.NaiveFileWALManager: Attempting to > recover c:\tmp\flume-ZHOUHH$\agent\zhouhh.TKOffice.local\sending / > 20130124-103034656+0800.2144885518902.00000028 > 13/01/25 15:31:09 INFO debug.MemorySinkSource: Opening mem source sink > that has with 0 elements > 13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Starting checksum > group called zhouhh.TKOffice.local2151099224995 > 13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: initial checksum is > 13c6a648fe8 > 13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Finishing checksum > group called 'zhouhh.TKOffice.local2151099224995' > 13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Checksum succeeded > 13c6a648fe8 > 13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: moved from partial to > complete zhouhh.TKOffice.local2151099224995 > Exception in thread "logicalNode zhouhh.TKOffice.local-25" > java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:59) > at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101) > at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1945) > at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1845) > at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1891) > at > com.cloudera.flume.handlers.hdfs.SeqfileEventSource.next(SeqfileEventSource.java:62) > at com.cloudera.flume.core.EventUtil.dumpAll(EventUtil.java:47) > at > com.cloudera.flume.agent.durability.NaiveFileWALManager.checkAndStripAckFraming(NaiveFileWALManager.java:378) > at > com.cloudera.flume.agent.durability.NaiveFileWALManager.recoverLog(NaiveFileWALManager.java:288) > at > com.cloudera.flume.agent.durability.NaiveFileWALManager.recover(NaiveFileWALManager.java:411) > at > com.cloudera.flume.agent.durability.NaiveFileWALDeco.open(NaiveFileWALDeco.java:240) > at com.cloudera.flume.agent.AgentSink.open(AgentSink.java:150) > at > com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88) > > > I have 4GB memory, and the source file to tail only 1KB, What can I do to > remove this error? > > Thanks! > Andy > >
