cgivre opened a new pull request, #3087: URL: https://github.com/apache/drill/pull/3087
# [DRILL-8556](https://issues.apache.org/jira/browse/DRILL-8556): Various PCAP and PCAP-NG Improvements ## Description The PCAP-NG reader read the whole file into memory (twice: the raw bytes plus the third-party library's list of every parsed block), so large captures needed several times their size in heap. This PR streams PCAP-NG files block by block, removes the `pcapngdecoder` dependency, and fixes a set of decoding bugs in both the PCAP and PCAP-NG readers. **PCAP-NG reader** - Streams blocks one at a time; memory no longer grows with file size. - Tracks interfaces per section: timestamps honor `if_tsresol` and `if_tsoffset` (nanosecond captures were off by 1000x), and packets are decoded for their interface's link type. - Decodes Ethernet (including 802.1Q/QinQ tags), raw IP, BSD loopback, PPP, Linux SLL/SLL2, and 802.11 / radiotap data frames. Unsupported link types return null IP fields instead of misread ones. - Skips unknown block types (previously it stopped silently at the first one) and tolerates a truncated final block. - Decodes each packet once per row instead of once per column. - Supports `sessionizeTCPStreams`. **Packet decoding (both readers)** - The TCP header length was read from the wrong byte and then over-counted, and payloads included link-layer headers, Ethernet padding, and bytes past the captured length. - IPv6: extension headers were walked incorrectly and TCP fields ignored the 40-byte IPv6 header, so IPv6 TCP sequence numbers, flags, and ports were misread. - `ARP_PROTOCOL` was defined as IP protocol 0 (IPv6 hop-by-hop), so IPv6 multicast packets were labeled ARP and real ARP frames were not. ARP is now detected by EtherType; ICMPv6 is labeled ICMP. **Format plugin** - The file format was a `static` field shared by all readers and was detected from the first file in the scan, so directories mixing `.pcap` and `.pcapng` sent files to the wrong reader. **TCP sessions (both readers)** - One shared `TcpSessionizer`; only TCP packets are tracked (all non-TCP packets previously accumulated in a "session 0" that was never written). - Sessions still open at end of file are returned with a new `session_closed` column instead of dropped. Packets trailing a closed session no longer start a phantom session. `connection_time` is null when the handshake was not captured. ## Documentation New PCAP-NG packet columns: `captured_length`, `interface_id`, `interface_name`, `link_type`, `comment`, `direction`, `reception_type`, `fcs_length`, `drop_count`, `packet_hash`. New session column: `session_closed`. In stat mode (`stat: true`), options a block does not have are now null (previously `""` or `-1`), a `comment` column is added, and `if_speed` / `if_tsoffset` are BIGINT. The plugin README documents the columns and supported link types. User-visible value changes from the bug fixes: classic PCAP `data` payloads, IPv6 TCP fields, `type` for ARP/ICMPv6, and `is_corrupt` (`testv1.pcap` reported 16 corrupt packets that were snaplen-truncated, not corrupt). ## Testing - All PCAP and PCAP-NG tests pass (46), plus checkstyle and the module `verify` build. - Replaced the corrupted `todo/` fixtures (they had been run through a UTF-8 conversion) with valid Wireshark captures, and added generated fixtures covering link types, timestamp resolutions, packet options, multiple sections, encapsulations and sessionization. Fixtures were cross-checked with scapy. - Every classic PCAP payload in the test fixtures was compared against scapy (6,736 exact matches; 17 encrypted payloads differ only in how invalid UTF-8 renders). Test expectations that encoded the old bugs were corrected after verifying the new values against scapy. Work in progress: a pluggable protocol decoder framework (`parsed_protocol` / `parsed_data` columns, DNS and HTTP first) will be added to this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
