Nick Allen created METRON-960:
---------------------------------
Summary: Pcap Topology - ArrayIndexOutOfBoundsException
Key: METRON-960
URL: https://issues.apache.org/jira/browse/METRON-960
Project: Metron
Issue Type: Bug
Reporter: Nick Allen
I am running the Pcap Topology with 128 executors and 128 tasks. It is
consuming packet data being landed into a Kafka topic with 128 partitions.
The topology has been running for at least 16 hours and I have been
sporadically sending it ~1 gbps of traffic for periods of roughly 5 - 30
minutes.
I am sending it new packets, its counters in the Storm UI are increasing, but
nothing is getting written to HDFS.
{code}
java.lang.ArrayIndexOutOfBoundsException: 4608 at
org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:76) at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:50)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:197) at
org.apache.hadoop.io.SequenceFile$Writer.sync(SequenceFile.java:1232) at
org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1293)
at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1333) at
org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1300) at
org.apache.metron.spout.pcap.PartitionHDFSWriter.handle(PartitionHDFSWriter.java:158)
at
org.apache.metron.spout.pcap.HDFSWriterCallback.apply(HDFSWriterCallback.java:149)
at org.apache.storm.kafka.CallbackCollector.emit(CallbackCollector.java:79) at
org.apache.storm.kafka.spout.KafkaSpout.emitTupleIfNotEmitted(KafkaSpout.java:342)
at org.apache.storm.kafka.spout.KafkaSpout.emit(KafkaSpout.java:307) at
org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:231) at
org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651)
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
{code}
I am also seeing other exception in the logs.
{code}
java.io.IOException: IOException flush:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
No lease on
/apps/metron/pcap/pcap_pcap128_1494962815458921000_118_pcap-62-1494971709
(inode 221567): File does not exist. [Lease. Holder:
DFSClient_NONMAPREDUCE_-100146495_67, pendingcreates: 9]
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3521)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.fsync(FSNamesystem.java:4120)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.fsync(NameNodeRpcServer.java:1334)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.fsync(ClientNamenodeProtocolServerSideTranslatorPB.java:911)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
at
org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:2050)
~[stormjar.jar:?]
at
org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1921)
~[stormjar.jar:?]
at
org.apache.hadoop.hdfs.client.HdfsDataOutputStream.hsync(HdfsDataOutputStream.java:95)
~[stormjar.jar:?]
at
org.apache.metron.spout.pcap.PartitionHDFSWriter$SyncHandlers$2.sync(PartitionHDFSWriter.java:71)
~[stormjar.jar:?]
at
org.apache.metron.spout.pcap.PartitionHDFSWriter.handle(PartitionHDFSWriter.java:161)
~[stormjar.jar:?]
at
org.apache.metron.spout.pcap.HDFSWriterCallback.apply(HDFSWriterCallback.java:149)
[stormjar.jar:?]
at
org.apache.storm.kafka.CallbackCollector.emit(CallbackCollector.java:79)
[stormjar.jar:?]
at
org.apache.storm.kafka.spout.KafkaSpout.emitTupleIfNotEmitted(KafkaSpout.java:342)
[stormjar.jar:?]
at org.apache.storm.kafka.spout.KafkaSpout.emit(KafkaSpout.java:307)
[stormjar.jar:?]
at
org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:231)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
