Hi Andy,
How about trying "refreshAll" command to see it helps?
It seems that logicalSource and logicalSink are not working perfectly
especially in finding and mapping logical node to physical node during
translation process in master node. I think "map Hadoop48 collect1"
should be done earlier so that master has mapping information for
collector1 to Hadoop48(physical node) so that translation can be done
properly when related configs were created.
for example,
(config) collect1: null | null
map Hadoop48 collect1
(config) collect1: logicalSource |
collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);
(config) agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
(config) agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
map Hadoop46 agent1
map Hadoop47 agent2
and, there seems typo in node name; you mapped "aggent1" to Hadoop46 but
name of the config you had created was "agent1"; it is why aggent1 on
Hadoop46 is "IDLE".
- JS
On 1/23/13 6:21 PM, ??? wrote:
hello,
I add logical config to flume:
agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
collect1: logicalSource |
collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);
map Hadoop46 agent1
map Hadoop47 agent2
map Hadoop48 collect1
============================
I run master on Hadoop48,
it reports:
2013-01-23 16:49:27,153 [main] INFO rolling.RollSink: Created
RollSink: trigger=[TimeTrigger: maxAge=5000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@7a960b08]
checkPeriodMs = 250
spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:27,158 [main] INFO rolling.RollSink: Created
RollSink: trigger=[TimeTrigger: maxAge=5000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1d29109e]
checkPeriodMs = 250
spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:27,207 [main] INFO master.ZooKeeperConfigStore:
Created new config at /flume-cfgs/cfg-0000000104
2013-01-23 16:49:27,207 [main-EventThread] INFO
master.ZooKeeperConfigStore: Config was updated - reloading
2013-01-23 16:49:27,255 [main] INFO master.ZooKeeperConfigStore:
Created new config at /flume-cfgs/cfg-0000000105
2013-01-23 16:49:27,255 [main-EventThread] INFO
master.ZooKeeperConfigStore: Config was updated - reloading
2013-01-23 16:49:27,307 [main] WARN
logical.LogicalConfigurationManager: Logical node 'collect1' has not
been assigned to a physical node yet
=================
run agent1 on Hadoop46, it reports:
2013-01-23 16:49:45,070 [Heartbeat] INFO rolling.RollSink: Created
RollSink: trigger=[TimeTrigger: maxAge=5000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1fd54f77]
checkPeriodMs = 250
spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:45,073 [Heartbeat] INFO agent.LogicalNodeManager:
creating new logical node collect1
2013-01-23 16:49:45,076 [logicalNode collect1-18] ERROR
connector.DirectDriver: Closing down due to exception on open calls
2013-01-23 16:49:45,076 [logicalNode collect1-18] INFO
connector.DirectDriver: Connector logicalNode collect1-18 exited with
error: Attempting to open a Stub Source!
2013-01-23 16:49:45,077 [logicalNode collect1-18] ERROR
connector.DirectDriver: Error closing logicalNode collect1-18 source:
Attempting to close a Stub Source!
2013-01-23 16:49:45,077 [logicalNode collect1-18] INFO
rolling.RollSink: closing RollSink
'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:45,081 [logicalNode collect1-18] INFO
rolling.RollSink: double close
'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
2013-01-23 16:49:45,081 [logicalNode collect1-18] ERROR
connector.DirectDriver: Exiting driver logicalNode collect1-18 in
error state StubSource | Collector because Attempting to open a Stub
Source!
2013-01-23 16:49:45,091 [Check config] INFO rolling.RollSink: Created
RollSink: trigger=[TimeTrigger: maxAge=5000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@5780208f]
checkPeriodMs = 250
spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
)'
===============
and agent2 on Hadoop47 , also logs error.
[zhouhh@Hadoop47 ~]$ flume node
2013-01-23 17:11:38,454 [main] INFO mortbay.log: Started
[email protected]:35862
<http://[email protected]:35862>
2013-01-23 17:11:38,454 [main] INFO util.InternalHttpServer: Server
started
2013-01-23 17:11:38,468 [Heartbeat] INFO agent.MultiMasterRPC: No
active master RPC connection
2013-01-23 17:11:38,495 [Heartbeat] INFO agent.ThriftMasterRPC:
Connected to master at hadoop48:35872
2013-01-23 17:11:38,687 [main] INFO agent.LogicalNodeManager: creating
new logical node Hadoop47
2013-01-23 17:11:38,819 [main] WARN agent.FlumeNode: Flume is using
Hadoop core 1.0.4 which does not support Security / Authentication: null
2013-01-23 17:11:43,529 [Heartbeat] INFO agent.LogicalNodeManager:
creating new logical node agent2
2013-01-23 17:11:43,536 [logicalNode agent2-18] ERROR
connector.DirectDriver: Closing down due to exception on open calls
2013-01-23 17:11:43,537 [logicalNode agent2-18] INFO
connector.DirectDriver: Connector logicalNode agent2-18 exited with
error: Attempting to open a stub sink 'fail'!
2013-01-23 17:11:43,537 [logicalNode agent2-18] ERROR
connector.DirectDriver: Exiting driver logicalNode agent2-18 in error
state TailSource | fail because Attempting to open a stub sink 'fail'!
2013-01-23 17:11:43,540 [TailThread-0] INFO text.TailSource:
TailThread has exited
2013-01-23 17:11:43,553 [Check config] INFO agent.LogicalNode: Node
config successfully set to
com.cloudera.flume.conf.FlumeConfigData@76a0f9cb
2013-01-23 17:11:43,554 [logicalNode agent2-20] ERROR
connector.DirectDriver: Closing down due to exception on open calls
2013-01-23 17:11:43,554 [logicalNode agent2-20] INFO
connector.DirectDriver: Connector logicalNode agent2-20 exited with
error: Attempting to open a stub sink 'fail'!
2013-01-23 17:11:43,554 [logicalNode agent2-20] ERROR
connector.DirectDriver: Exiting driver logicalNode agent2-20 in error
state TailSource | fail because Attempting to open a stub sink 'fail'!
2013-01-23 17:11:43,554 [TailThread-1] INFO text.TailSource:
TailThread has exited
http://hadoop48:35871/flumemaster.jsp
Node status
logical node physical node host name status version last seen
delta (s) last seen
Hadoop46 Hadoop46 Hadoop46 IDLE none 2 Wed Jan
23 17:07:05 CST 2013
Hadoop47 Hadoop47 Hadoop47 IDLE none 1 Wed Jan
23 17:07:06 CST 2013
Hadoop48 Hadoop48 Hadoop48 IDLE Wed Jan 23 16:49:27 CST 2013 0
Wed Jan 23 17:07:08 CST 2013
WX-wuwb.TKOffice.local WX-wuwb.TKOffice.local
WX-wuwb.TKOffice.local ERROR Wed Jan 23 16:49:27 CST 2013 0 Wed
Jan 23 17:07:07 CST 2013
agent2 Hadoop47 Hadoop47 ERROR Wed Jan 23 16:49:27 CST 2013 1
Wed Jan 23 17:07:06 CST 2013
aggent1 Hadoop46 Hadoop46 IDLE none 2 Wed Jan
23 17:07:05 CST 2013
co1 co1 Hadoop48 LOST Wed Jan 23 09:56:48 CST 2013 1057 Wed Jan
23 16:49:30 CST 2013
collect1 Hadoop48 Hadoop48 ERROR Wed Jan 23 16:49:27 CST 2013 0
Wed Jan 23 17:07:08 CST 2013
zhouhh.TKOffice.local zhouhh.TKOffice.local
zhouhh.TKOffice.local IDLE Wed Jan 23 16:49:27 CST 2013 0 Wed Jan
23 17:07:07 CST 2013
Physical/Logical Node mapping
physical node logical node
ag1 ag1
WX-wuwb.TKOffice.local WX-wuwb.TKOffice.local
hadoop48conn hadoop48conn
ag46 ag46
zhouhh.TKOffice.local zhouhh.TKOffice.local
ag47 ag47
co1 co1
Hadoop47 Hadoop47,agent2
Hadoop48 Hadoop48,collect1
collector collector
h48co2 h48co2
h48coll h48coll
Hadoop46 Hadoop46,aggent1
some one can tell me what's matter?
thanks!
Andy
--
Jeong-shik Jang / [email protected]
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting
