[
https://issues.apache.org/jira/browse/DRILL-8556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18122056#comment-18122056
]
ASF GitHub Bot commented on DRILL-8556:
---------------------------------------
cgivre opened a new pull request, #3087:
URL: https://github.com/apache/drill/pull/3087
# [DRILL-8556](https://issues.apache.org/jira/browse/DRILL-8556): Various
PCAP and PCAP-NG Improvements
## Description
The PCAP-NG reader read the whole file into memory (twice: the raw bytes
plus the third-party library's list of every parsed block), so large captures
needed several times their size in heap. This PR streams PCAP-NG files block by
block, removes the `pcapngdecoder` dependency, and fixes a set of decoding bugs
in both the PCAP and PCAP-NG readers.
**PCAP-NG reader**
- Streams blocks one at a time; memory no longer grows with file size.
- Tracks interfaces per section: timestamps honor `if_tsresol` and
`if_tsoffset` (nanosecond captures were off by 1000x), and packets are decoded
for their interface's link type.
- Decodes Ethernet (including 802.1Q/QinQ tags), raw IP, BSD loopback, PPP,
Linux SLL/SLL2, and 802.11 / radiotap data frames. Unsupported link types
return null IP fields instead of misread ones.
- Skips unknown block types (previously it stopped silently at the first
one) and tolerates a truncated final block.
- Decodes each packet once per row instead of once per column.
- Supports `sessionizeTCPStreams`.
**Packet decoding (both readers)**
- The TCP header length was read from the wrong byte and then over-counted,
and payloads included link-layer headers, Ethernet padding, and bytes past the
captured length.
- IPv6: extension headers were walked incorrectly and TCP fields ignored the
40-byte IPv6 header, so IPv6 TCP sequence numbers, flags, and ports were
misread.
- `ARP_PROTOCOL` was defined as IP protocol 0 (IPv6 hop-by-hop), so IPv6
multicast packets were labeled ARP and real ARP frames were not. ARP is now
detected by EtherType; ICMPv6 is labeled ICMP.
**Format plugin**
- The file format was a `static` field shared by all readers and was
detected from the first file in the scan, so directories mixing `.pcap` and
`.pcapng` sent files to the wrong reader.
**TCP sessions (both readers)**
- One shared `TcpSessionizer`; only TCP packets are tracked (all non-TCP
packets previously accumulated in a "session 0" that was never written).
- Sessions still open at end of file are returned with a new
`session_closed` column instead of dropped. Packets trailing a closed session
no longer start a phantom session. `connection_time` is null when the handshake
was not captured.
## Documentation
New PCAP-NG packet columns: `captured_length`, `interface_id`,
`interface_name`, `link_type`, `comment`, `direction`, `reception_type`,
`fcs_length`, `drop_count`, `packet_hash`. New session column:
`session_closed`. In stat mode (`stat: true`), options a block does not have
are now null (previously `""` or `-1`), a `comment` column is added, and
`if_speed` / `if_tsoffset` are BIGINT. The plugin README documents the columns
and supported link types.
User-visible value changes from the bug fixes: classic PCAP `data` payloads,
IPv6 TCP fields, `type` for ARP/ICMPv6, and `is_corrupt` (`testv1.pcap`
reported 16 corrupt packets that were snaplen-truncated, not corrupt).
## Testing
- All PCAP and PCAP-NG tests pass (46), plus checkstyle and the module
`verify` build.
- Replaced the corrupted `todo/` fixtures (they had been run through a UTF-8
conversion) with valid Wireshark captures, and added generated fixtures
covering link types, timestamp resolutions, packet options, multiple sections,
encapsulations and sessionization. Fixtures were cross-checked with scapy.
- Every classic PCAP payload in the test fixtures was compared against scapy
(6,736 exact matches; 17 encrypted payloads differ only in how invalid UTF-8
renders). Test expectations that encoded the old bugs were corrected after
verifying the new values against scapy.
Work in progress: a pluggable protocol decoder framework (`parsed_protocol`
/ `parsed_data` columns, DNS and HTTP first) will be added to this PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
> Various PCAP and PCAP-NG Improvements
> -------------------------------------
>
> Key: DRILL-8556
> URL: https://issues.apache.org/jira/browse/DRILL-8556
> Project: Apache Drill
> Issue Type: Improvement
> Components: Format - PCAP
> Affects Versions: 1.22.0
> Reporter: Charles Givre
> Assignee: Charles Givre
> Priority: Major
> Fix For: 1.23.0
>
>
> See PR for full details. Â This PR contains several fixes and performance
> enhancements to the PCAP reader.Â
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
