[
https://issues.apache.org/jira/browse/METRON-960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014092#comment-16014092
]
Nick Allen edited comment on METRON-960 at 5/17/17 2:03 PM:
------------------------------------------------------------
The exception does seem to complain that the file does not exist.
{code}
[metron@y136 ~]$ hdfs dfs -ls
/apps/metron/pcap/pcap_pcap128_1494962815458921000_118_pcap-62-1494971709
-rw-r--r-- 1 metron hadoop 2859704978 2017-05-17 13:17
/apps/metron/pcap/pcap_pcap128_1494962815458921000_118_pcap-62-1494971709
{code}
was (Author: nickwallen):
The exception does seem to complain that the file does not exist.
```
[metron@y136 ~]$ hdfs dfs -ls
/apps/metron/pcap/pcap_pcap128_1494962815458921000_118_pcap-62-1494971709
-rw-r--r-- 1 metron hadoop 2859704978 2017-05-17 13:17
/apps/metron/pcap/pcap_pcap128_1494962815458921000_118_pcap-62-1494971709
```
> Pcap Topology - ArrayIndexOutOfBoundsException
> ----------------------------------------------
>
> Key: METRON-960
> URL: https://issues.apache.org/jira/browse/METRON-960
> Project: Metron
> Issue Type: Bug
> Reporter: Nick Allen
>
> I am running the Pcap Topology with 128 executors and 128 tasks. It is
> consuming packet data being landed into a Kafka topic with 128 partitions.
> The topology has been running for at least 16 hours and I have been
> sporadically sending it ~1 gbps of traffic for periods of roughly 5 - 30
> minutes.
> I am sending it new packets, its counters in the Storm UI are increasing, but
> nothing is getting written to HDFS.
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 4608 at
> org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:76) at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:50)
> at java.io.DataOutputStream.writeInt(DataOutputStream.java:197) at
> org.apache.hadoop.io.SequenceFile$Writer.sync(SequenceFile.java:1232) at
> org.apache.hadoop.io.SequenceFile$Writer.checkAndWriteSync(SequenceFile.java:1293)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1333)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1300) at
> org.apache.metron.spout.pcap.PartitionHDFSWriter.handle(PartitionHDFSWriter.java:158)
> at
> org.apache.metron.spout.pcap.HDFSWriterCallback.apply(HDFSWriterCallback.java:149)
> at org.apache.storm.kafka.CallbackCollector.emit(CallbackCollector.java:79)
> at
> org.apache.storm.kafka.spout.KafkaSpout.emitTupleIfNotEmitted(KafkaSpout.java:342)
> at org.apache.storm.kafka.spout.KafkaSpout.emit(KafkaSpout.java:307) at
> org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:231) at
> org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651)
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
> clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
> {code}
> I am also seeing other exception in the logs.
> {code}
> java.io.IOException: IOException flush:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> No lease on
> /apps/metron/pcap/pcap_pcap128_1494962815458921000_118_pcap-62-1494971709
> (inode 221567): File does not exist. [Lease. Holder:
> DFSClient_NONMAPREDUCE_-100146495_67, pendingcreates: 9]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3521)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.fsync(FSNamesystem.java:4120)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.fsync(NameNodeRpcServer.java:1334)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.fsync(ClientNamenodeProtocolServerSideTranslatorPB.java:911)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:2050)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1921)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hdfs.client.HdfsDataOutputStream.hsync(HdfsDataOutputStream.java:95)
> ~[stormjar.jar:?]
> at
> org.apache.metron.spout.pcap.PartitionHDFSWriter$SyncHandlers$2.sync(PartitionHDFSWriter.java:71)
> ~[stormjar.jar:?]
> at
> org.apache.metron.spout.pcap.PartitionHDFSWriter.handle(PartitionHDFSWriter.java:161)
> ~[stormjar.jar:?]
> at
> org.apache.metron.spout.pcap.HDFSWriterCallback.apply(HDFSWriterCallback.java:149)
> [stormjar.jar:?]
> at
> org.apache.storm.kafka.CallbackCollector.emit(CallbackCollector.java:79)
> [stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.KafkaSpout.emitTupleIfNotEmitted(KafkaSpout.java:342)
> [stormjar.jar:?]
> at org.apache.storm.kafka.spout.KafkaSpout.emit(KafkaSpout.java:307)
> [stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:231)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651)
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
