[
https://issues.apache.org/jira/browse/EAGLE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15043788#comment-15043788
]
Hao Chen commented on EAGLE-2:
------------------------------
{code}
Exception in thread "main" java.lang.UnsupportedOperationException: output
fields should be declared in sub class of KafkaSourcedSpoutProvider
at
org.apache.eagle.dataproc.impl.storm.kafka.KafkaSourcedSpoutScheme.getOutputFields(KafkaSourcedSpoutScheme.java:63)
at
backtype.storm.spout.SchemeAsMultiScheme.getOutputFields(SchemeAsMultiScheme.java:39)
at storm.kafka.KafkaSpout.declareOutputFields(KafkaSpout.java:185)
at
backtype.storm.topology.TopologyBuilder.getComponentCommon(TopologyBuilder.java:226)
at
backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:110)
at
org.apache.eagle.datastream.StormTopologyCompiler.buildTopology(StormTopologyCompiler.scala:91)
at
org.apache.eagle.datastream.StormExecutionEnvironment.execute(ExecutionEnvironment.scala:96)
at
org.apache.eagle.datastream.ExecutionEnvironmentBase.execute(ExecutionEnvironment.scala:88)
at
org.apache.eagle.datastream.kafka.KafkaStreamMonitorApp$delayedInit$body.apply(KafkaStreamMonitor.scala:30)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:309)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at
org.apache.eagle.datastream.kafka.KafkaStreamMonitorApp.main(KafkaStreamMonitor.scala:22)
at
org.apache.eagle.datastream.TestKafkaStreamMonitor.main(TestKafkaStreamMonitor.java:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
{code}
> watch message process backlog in Eagle UI
> -----------------------------------------
>
> Key: EAGLE-2
> URL: https://issues.apache.org/jira/browse/EAGLE-2
> Project: Eagle
> Issue Type: Improvement
> Environment: production
> Reporter: Edward Zhang
> Assignee: Libin, Sun
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Message latency is a key factor for Eagle to enable realtime security
> monitoring. For hdfs audit log monitoring, kafka is used as datasource. So
> there is always some gap between current max offset in kafka and processed
> offset in eagle. The gap is the backlog which eagle should consume quickly as
> much as quickly. If the gap can be sampled for every minute or 20 seconds,
> then we understand if eagle is catching up or is lagging behind more.
> The command to get current max offset in kafka is
> bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list xxxx --topic
> hdfs_audit_log --time -1
> and Storm-kafka spout would store processed offset in zookeeper, in the
> following znode:
> /consumers/hdfs_audit_log/eagle.hdfsaudit.consumer/partition_0
> So technically we can get the gap and write that to eagle service then in UI
> we can watch the backlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
