[
https://issues.apache.org/jira/browse/METRON-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Stella updated METRON-443:
--------------------------------
Assignee: Justin Leet (was: Casey Stella)
> Exception seen while running stellar or fixed query on pcap data
> ----------------------------------------------------------------
>
> Key: METRON-443
> URL: https://issues.apache.org/jira/browse/METRON-443
> Project: Metron
> Issue Type: Bug
> Affects Versions: 0.2.2BETA
> Reporter: Neha Sinha
> Assignee: Justin Leet
> Fix For: 0.2.2BETA
>
>
> I am noticing the following exception once in a while when i run the pcap
> stellar/fixed query.
> Console Log error
> =========================================================
> root@metron-multi-3 ~# /usr/metron/0.2.0BETA/bin/pcap_query.sh fixed -st
> "20160726" -df "yyyyMMdd" -sa "192.168.138.158" -da "62.75.195.236" -sp
> "49184" -dp "80" -p "6"
> 16/07/29 06:50:54 INFO impl.TimelineClientImpl: Timeline service address:
> http://metron-multi-8.openstacklocal:8188/ws/v1/timeline/
> 16/07/29 06:50:54 INFO client.RMProxy: Connecting to ResourceManager at
> metron-multi-2.openstacklocal/172.22.104.43:8050
> 16/07/29 06:50:56 INFO input.FileInputFormat: Total input paths to process :
> 17
> 16/07/29 06:50:56 INFO mapreduce.JobSubmitter: number of splits:17
> 16/07/29 06:50:57 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1469678414974_0019
> 16/07/29 06:50:57 INFO impl.YarnClientImpl: Submitted application
> application_1469678414974_0019
> 16/07/29 06:50:57 INFO mapreduce.Job: The url to track the job:
> http://metron-multi-2.openstacklocal:8088/proxy/application_1469678414974_0019/
> 16/07/29 06:50:57 INFO mapreduce.Job: Running job: job_1469678414974_0019
> 16/07/29 06:51:09 INFO mapreduce.Job: Job job_1469678414974_0019 running in
> uber mode : false
> 16/07/29 06:51:09 INFO mapreduce.Job: map 0% reduce 0%
> 16/07/29 06:51:19 INFO mapreduce.Job: map 12% reduce 0%
> 16/07/29 06:51:22 INFO mapreduce.Job: map 24% reduce 0%
> 16/07/29 06:51:23 INFO mapreduce.Job: map 41% reduce 0%
> 16/07/29 06:51:24 INFO mapreduce.Job: map 53% reduce 0%
> 16/07/29 06:51:25 INFO mapreduce.Job: map 59% reduce 0%
> 16/07/29 06:51:26 INFO mapreduce.Job: map 71% reduce 0%
> 16/07/29 06:51:30 INFO mapreduce.Job: map 76% reduce 0%
> 16/07/29 06:51:32 INFO mapreduce.Job: map 82% reduce 0%
> 16/07/29 06:51:34 INFO mapreduce.Job: map 88% reduce 0%
> 16/07/29 06:51:35 INFO mapreduce.Job: map 100% reduce 0%
> 16/07/29 06:51:36 INFO mapreduce.Job: map 100% reduce 100%
> 16/07/29 06:51:36 INFO mapreduce.Job: Job job_1469678414974_0019 completed
> successfully
> 16/07/29 06:51:36 INFO mapreduce.Job: Counters: 50
> File System Counters
> FILE: Number of bytes read=60586
> FILE: Number of bytes written=2582180
> FILE: Number of read operations=0
> FILE: Number of large read operations=0
> FILE: Number of write operations=0
> HDFS: Number of bytes read=1936634
> HDFS: Number of bytes written=62671
> HDFS: Number of read operations=71
> HDFS: Number of large read operations=0
> HDFS: Number of write operations=2
> Job Counters
> Launched map tasks=17
> Launched reduce tasks=1
> Data-local map tasks=16
> Rack-local map tasks=1
> Total time spent by all maps in occupied slots (ms)=489258
> Total time spent by all reduces in occupied slots (ms)=40128
> Total time spent by all map tasks (ms)=163086
> Total time spent by all reduce tasks (ms)=13376
> Total vcore-seconds taken by all map tasks=163086
> Total vcore-seconds taken by all reduce tasks=13376
> Total megabyte-seconds taken by all map tasks=200432694
> Total megabyte-seconds taken by all reduce tasks=16439104
> Map-Reduce Framework
> Map input records=2971
> Map output records=260
> Map output bytes=59956
> Map output materialized bytes=60682
> Input split bytes=3264
> Combine input records=0
> Combine output records=0
> Reduce input groups=95
> Reduce shuffle bytes=60682
> Reduce input records=260
> Reduce output records=260
> Spilled Records=520
> Shuffled Maps =17
> Failed Shuffles=0
> Merged Map outputs=17
> GC time elapsed (ms)=4115
> CPU time spent (ms)=19170
> Physical memory (bytes) snapshot=8603783168
> Virtual memory (bytes) snapshot=42252124160
> Total committed heap usage (bytes)=6922698752
> Shuffle Errors
> BAD_ID=0
> CONNECTION=0
> IO_ERROR=0
> WRONG_LENGTH=0
> WRONG_MAP=0
> WRONG_REDUCE=0
> File Input Format Counters
> Bytes Read=1933370
> File Output Format Counters
> Bytes Written=62671
> Exception in thread "main" java.lang.IllegalArgumentException: Comparison
> method violates its general contract!
> at java.util.TimSort.mergeHi(TimSort.java:895)
> at java.util.TimSort.mergeAt(TimSort.java:512)
> at java.util.TimSort.mergeCollapse(TimSort.java:435)
> at java.util.TimSort.sort(TimSort.java:241)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at org.apache.metron.pcap.PcapMerger.sort(PcapMerger.java:165)
> at org.apache.metron.pcap.PcapMerger.merge(PcapMerger.java:110)
> at
> org.apache.metron.pcap.query.ResultsWriter.mergePcaps(ResultsWriter.java:45)
> at org.apache.metron.pcap.query.ResultsWriter.write(ResultsWriter.java:33)
> at org.apache.metron.pcap.query.PcapCli.run(PcapCli.java:149)
> at org.apache.metron.pcap.query.PcapCli.main(PcapCli.java:46)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> ========================================================
> PCAP Contents
> ========================================================
> root@metron-multi-3 ~# /usr/metron/0.2.0BETA/bin/pcap_inspector.sh -i
> /apps/metron/pcap/pcap_pcap_1469521273134478000_0_e05b4319-6498-4dce-bee2-02f0e1e28755
> -n 5
> TS: July 26, 2016 8:21:13 AM UTC,ip_src_addr: 172.22.80.156,ip_src_port:
> 44324,ip_dst_addr: 172.22.80.228,ip_dst_port: 6667,protocol: 6
> TS: July 26, 2016 8:21:13 AM UTC,ip_src_addr: 172.22.80.156,ip_src_port:
> 44324,ip_dst_addr: 172.22.80.228,ip_dst_port: 6667,protocol: 6
> TS: July 26, 2016 8:21:13 AM UTC,ip_src_addr: 172.22.80.156,ip_src_port:
> 44324,ip_dst_addr: 172.22.80.228,ip_dst_port: 6667,protocol: 6
> TS: July 26, 2016 8:21:13 AM UTC,ip_src_addr: 172.22.80.228,ip_src_port:
> 6667,ip_dst_addr: 172.22.80.156,ip_dst_port: 44324,protocol: 6
> TS: July 26, 2016 8:21:13 AM UTC,ip_src_addr: 172.22.80.156,ip_src_port:
> 44324,ip_dst_addr: 172.22.80.228,ip_dst_port: 6667,protocol: 6
> ========================================================
> Fixed Query
> ========================================================
> /usr/metron/0.2.0BETA/bin/pcap_query.sh fixed -st "20160726" -df "yyyyMMdd"
> -sa "172.22.80.156" -da "172.22.80.228" -sp "6667" -dp "44324" -p “6"
> ========================================================
> Stellar Query
> ========================================================
> /usr/metron/0.2.0BETA/bin/pcap_query.sh query -st "20160726" -df "yyyyMMdd"
> -query "ip_src_addr == '172.22.80.156' and ip_dst_addr == '172.22.80.228' and
> ip_src_port == '6667' ip_dst_port == '4432' and protocol== '6'"
> ========================================================
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
