Hi,
I want gather different format and name log file from agent. I want to
write each file to hdfs with different file name prefix or different dir so
that I can recognize the files are from which.
source like:
config [MAgent-44, text("D:\\TKServer\\_BakLog\\20130207 655game.log"),
batch(1000) agentDFOSink("192.168.10.48", 35853)]
config [MAgent-44, text("D:\\TKServer\\_BakLog\\20130207 655user.log"),
batch(1000) agentDFOSink("192.168.10.48", 35853)]
config [MAgent-45, text("D:\\TKServer\\_BakLog\\20130207 655game.log"),
batch(1000) agentDFOSink("192.168.10.48", 35853)]
config [MAgent-45, text("D:\\TKServer\\_BakLog\\20130207 655user.log"),
batch(1000) agentDFOSink("192.168.10.48", 35853)]
collector like
config [co1, collectorSource( 35853 ), gunzip unbatch collectorSink(
"hdfs://hadoop48:54310/user/flume/%y%m/%d","%{host}-%{sourcefile}-")]
note:%{sourcefile} isn't existed.
results like:
-rw-r--r-- 2 zhouhh supergroup 7309058 2013-02-07 16:14
/user/flume/1302/07/MAgent-44-game.log-20130207-161423054+0800.885170506522053.00035553
-rw-r--r-- 2 zhouhh supergroup 17922102 2013-02-07 16:14
/user/flume/1302/07/MAgent-44-user.log-20130207-161453158+0800.885200610609053.00035551
-rw-r--r-- 2 zhouhh supergroup 17854942 2013-02-07 16:15
/user/flume/1302/07/MAgent-45-game.log-20130207-161523249+0800.885230701184053.00035551
-rw-r--r-- 2 zhouhh supergroup 17827087 2013-02-07 16:15
/user/flume/1302/07/MAgent-45-user.log-20130207-161553269+0800.885260721933053.00035551
-rw-r--r-- 2 zhouhh supergroup 17820650 2013-02-07 16:16
/user/flume/1302/07/MAgent-44-game.log-20130207-161623290+0800.885290742023053.00035551
how can I do this?
can I using different ports of collector to map different source files?
Thanks.
Andy
