Hi Ted, EVF = Enhanced Vector Framework. Complete tutorial here: https://github.com/paul-rogers/drill/wiki/Developer%27s-Guide-to-the-Enhanced-Vector-Framework#basics-tutorial <https://github.com/paul-rogers/drill/wiki/Developer's-Guide-to-the-Enhanced-Vector-Framework#basics-tutorial> Basically, what I was thinking was that we can use the EVF to define the schema for known columns (IE level 1 & 2 headers). EVF handles pushdown projection so we could eliminate a lot of that logic in the plugin. Then EVF also allows dynamic schema discovery, so we could create a map called packet_data or whatever, and that would be populated with whatever fields exist in the packet. We would need to write or otherwise obtain protocol dissectors for the different protocols but I'm going to start wtih DNS since I need that for work. I'm pretty sure that the EVF allows for variant maps so if you have a DNS packet and a ICMP packet, you'd get different fields in the map. -- C
> On Sep 22, 2019, at 11:30 AM, Ted Dunning <[email protected]> wrote: > > This sounds amazing. > > Some questions. > > What is EVF? > > How can you deal with the problem of variant maps? > > On Sun, Sep 22, 2019, 7:55 AM Charles Givre <[email protected]> wrote: > >> Hello all, >> I'm contemplating some improvements to Drill's PCAP reader. Specifically, >> I'd like for Drill to actually be able to parse some of the actual packet >> data. I was thinking of using KaiTai structs as a means to do so as they >> already have parsers for common packets. An example of this is the DNS >> parser (https://formats.kaitai.io/dns_packet/java.html) >> >> I was thinking of doing the following: >> 1. Converting the PCAP plugin to use the EVF framework. >> 2. Including a config option to turn the parsing on/off >> 3. Having the appropriate parser read and parse the data and store it >> into a Drill map. >> >> Does anyone have any comments or thoughts on the matter? >> Thanks, >> -- C >> >>
