The pcap format is there almost entirely just to give us a record structure.
The stream could give us that just as easily. So that is one thing that could be done. Another thought is that data could be buffered a (very) short time and groups of packets could be sent as a single message. That would decrease streaming overhead. I would vote against that change at first, but it would make the pcap format relevant again. On Tue, Jan 9, 2018 at 6:45 AM, John Omernik <[email protected]> wrote: > I've been following this and and also now playing with them more. A couple > of tidbits that I think are worth re-mentioning: > > 1. pcapng the ability to read this too will be handy. > > 2. The flags are really cool, and the work here is great. > > The other thing I'd like to consider for this is with recent work on the > Kafka plugin, and I am guessing a similar plugin with MapR Streams, what if > we were able to have a tool like daemonlogger, or tcpdump read from a > network interface and write directly to a Kafka/MapR Stream topic? > > My thought is this, you write to the topic, you partition based on a hash > of known data components (like the 5-tuple of protocol, sport, dport, > saddr, and daddr). From here, you could have things subscribe to the topic > and process, you could etl, you could query. There would be some concern > at high scale, but even that has methods to address. One of the challenges > we have infosec is getting processing close enough to capture.... OR > capture to file, move them around, have creates process to read and process > files. > > It would be cool, to combine the pcap plugin directly with the kafka/stream > plugin in drill and be able to read live pcaps! > > One big challenge is the pcap format has a header, but it seems to be a > simple header that we "may" be able to forge upon dropping the needle on a > stream of packet records. I am just brainstorming aloud right now, (I > should consider waiting until at least 3 cups of coffee before I make this > inner dialog public) > > Curious on thoughts here. > > > > > On Fri, Jan 5, 2018 at 10:01 AM, Charles Givre <[email protected]> wrote: > > > I assumed (incorrectly as it turned out) that I was not reading the > > offsets correctly or something. > > > > > > > On Jan 5, 2018, at 10:59, Ted Dunning <[email protected]> wrote: > > > > > > Yeah... I got the same result. > > > > > > I will push my latest shortly (without the boolean stuff) > > > > > > On Fri, Jan 5, 2018 at 7:52 AM, Charles Givre <[email protected]> > wrote: > > > > > >> Ted, > > >> I’m wondering if we’ve uncovered an unsupported, undocumented feature > in > > >> Drill with respect to the BitWriter. I made the following changes: > > >> > > >> > > >> To PcapRecordReader.java, I added: > > >> > > >> > > >> case "ack_flag": > > >> if (packet.isTcpPacket()) { > > >> int flag = (packet.getAckFlag()) ? 1 : 0; > > >> setBooleanColumnValue(flag, pci, count); > > >> } > > >> break; > > >> ... > > >> > > >> private void setBooleanColumnValue(final int data, final > > >> ProjectedColumnInfo pci, final int count) { > > >> ((NullableBitVector.Mutator) pci.vv.getMutator()) > > >> .setSafe(count, data); > > >> } > > >> > > >> Then I added this to PcapFormatUtils > > >> > > >> public static int getBit(final byte[] buf, final int posByte, final > int > > >> posBit ) { > > >> byte valByte = buf[posByte]; > > >> return valByte >> (8 - (posBit + 1)) & 0x0001; > > >> } > > >> > > >> I added a column in the Schema.java and a function in Packet.java to > get > > >> the bit. I was getting NPEs and I’m wondering now if the cause wasn’t > > my > > >> code but rather a problem with the bitwriter. I’m going to play with > > this > > >> and see if I can get Drill to write a True/False value at all, and > > report > > >> back. If the BitWriter is throwing NPE, I’ll create a JIRA for it. > > >> > > >> Thanks for your work on the PCAP format reader. > > >> —C > > >> > > >> > > >> > > >>> On Jan 3, 2018, at 17:33, Ted Dunning <[email protected]> wrote: > > >>> > > >>> Don't think that will work. > > >>> > > >>> I tried what seemed obvious and got a NPE. Joys of Drill. > > >>> > > >>> > > >>> > > >>> On Wed, Jan 3, 2018 at 1:31 PM, Charles Givre <[email protected]> > > wrote: > > >>> > > >>>> This isn’t the most elegant example, but should do the trick. > > >>>> > > >>>> https://github.com/cgivre/drill-network-functions/blob/ > > >>>> master/src/main/java/org/apache/drill/contrib/function/ > > IsPrivateIP.java > > >> < > > >>>> https://github.com/cgivre/drill-network-functions/blob/ > > >>>> master/src/main/java/org/apache/drill/contrib/function/ > > >> IsPrivateIP.java> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>>> On Jan 3, 2018, at 16:09, Ted Dunning <[email protected]> > wrote: > > >>>>> > > >>>>> On Wed, Jan 3, 2018 at 12:17 PM, Charles Givre <[email protected]> > > >> wrote: > > >>>>> > > >>>>>> HI Ted, > > >>>>>> This is really looking amazing. Thank you so much for working on > > >>>> this. I > > >>>>>> wanted to ask whether you’ve tried the BitWriter to write a > Boolean > > >>>> value? > > >>>>>> I’ve done that in UDFs. > > >>>>> > > >>>>> > > >>>>> Point me to an example. I think booleans would be vastly better. > > >>>> > > >>>> > > >> > > >> > > > > >
