[ https://issues.apache.org/jira/browse/DRILL-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16031564#comment-16031564 ]
ASF GitHub Bot commented on DRILL-5432: --------------------------------------- Github user parthchandra commented on the issue: https://github.com/apache/drill/pull/831 The contrib directory is where we have, in the past, added new storage and format plugins that are new and may not have been sufficiently tested. For this plugin, I think testing with pcap files from different sources would be useful. [1,2] are useful sources for data that will test boundary conditions. I tried on a file from [2] and got an NPE (didn't investigate the cause). A random sample of files from [1] worked very nicely indeed, though I didn't validate the output. You might have already done this level of testing; if so, I will withdraw the suggestion. [1] https://wiki.wireshark.org/SampleCaptures#Captures_used_in_Wireshark_testing [2] http://www.netresec.com/?page=PcapFiles > Want a memory format for PCAP files > ----------------------------------- > > Key: DRILL-5432 > URL: https://issues.apache.org/jira/browse/DRILL-5432 > Project: Apache Drill > Issue Type: New Feature > Reporter: Ted Dunning > > PCAP files [1] are the de facto standard for storing network capture data. In > security and protocol applications, it is very common to want to extract > particular packets from a capture for further analysis. > At a first level, it is desirable to query and filter by source and > destination IP and port or by protocol. Beyond that, however, it would be > very useful to be able to group packets by TCP session and eventually to look > at packet contents. For now, however, the most critical requirement is that > we should be able to scan captures at very high speed. > I previously wrote a (kind of working) proof of concept for a PCAP decoder > that did lazy deserialization and could traverse hundreds of MB of PCAP data > per second per core. This compares to roughly 2-3 MB/s for widely available > Apache-compatible open source PCAP decoders. > This JIRA covers the integration and extension of that proof of concept as a > Drill file format. > Initial work is available at https://github.com/mapr-demos/drill-pcap-format > [1] https://en.wikipedia.org/wiki/Pcap -- This message was sent by Atlassian JIRA (v6.3.15#6346)