Evan Huus skrev 2013-08-28 00:17:
On Tue, Aug 27, 2013 at 6:02 PM, Anders Broman <[email protected]
<mailto:[email protected]>> wrote:
Joerg Mayer skrev 2013-08-27 23:24:
On Tue, Aug 27, 2013 at 05:09:19PM -0400, Evan Huus wrote:
IIRC, two-pass allows for most/all of the
reassembly/request-response
stuff,
which we want to do sometimes. Any ideas why we have
to keep some
information
indefinitely?
Two-pass requires us to keep *all* the state around
through the first pass
so that it is available during the second pass (at which
point it can be
discarded). Even in single-pass mode, there is some state
that we can't
always immediately discard. If I see a fragment of a TCP
message then it
doesn't make sense to discard that until the other
fragments have arrived
and been reassembled. If I see a request, I probably need
to keep state
from that request until the response (which may never show
up).
We already do reassembly and a lot of other stateful work
in single-pass
mode. The only thing two-pass mode provides is the ability
to "see the
future" (for example, saying: this request has a response
5 packets later).
So (assuming we really free everything we could already) could
add a
possibly configurable foresight horizon of 10000 packets. If a
packet
number is older than 10000 packets, forget it?
I haven't really looked but we do keep the reassembled fragments
around even in tshark as there is no
mechanism to discard them selectively if run by tshark as opposed
to wireshark and that's the really big memory eater I would think.
This is the primary problem. Any state saved with p_add_proto_data, or
the reassembly API, or the conversation API, or (god-forbid) globals
is not being freed right now. Any memory allocated with seasonal/file
scope is not being freed right now. We do free frame data, column data
and misc. other bits and pieces, but the majority of the state we
create is not freed until the end of the file.
As Anders says, this is because we have no way right now to
selectively discard it: much of the data is stored in a way that we
can only get rid of all of it, or none. Part of that is because there
was no se_free call (for which wmem_free is the convenient solution)
and part of that is just because nobody's ever added it to, for
example, the reassembly API.
I'm sure there are some significant improvements we could make if
somebody figures out how, but you also have to beware of bugs like
9027 [1] which is occurring precisely because we were trying to free
old reassembly data. It turns out there are certain cases where that
data, at least, could still be in use. The logic to selectively free
state is not simple, and is often dissector-dependent.
Cheers,
Evan
[1] https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=9027
P.S. Given the above it would actually be fairly easy to create a
fully state-less tshark: make file-scope memory redirect to
packet-scope, and make our stateful APIs (p_add_proto_data,
reassembly, conversations) no-ops. I'm sure this will break some
dissectors that use globals, but they should be fixed anyways :)
>make our stateful APIs (p_add_proto_data, reassembly, conversations)
no-ops
I'm not sure I get this, p_add_proto_data might work as a no-op in
tshark's case but isn't conversation data
reassembly, needed to fully dissect a "future" frame?
A way to fix the reassembly case might be to store the fully reassembled
data with p_add_proto_data rather than
in a hash_table and in tshark's case discard it when finished with the
frame this might have other benefits as well.
Unfortunately it means changing all dissectors using reassembly but if
the most frequently used ones
where done it might fix most use cases ( TCP, HTTP...?).
___________________________________________________________________________
Sent via: Wireshark-dev mailing list <[email protected]>
Archives: http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
mailto:[email protected]?subject=unsubscribe
___________________________________________________________________________
Sent via: Wireshark-dev mailing list <[email protected]>
Archives: http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
mailto:[email protected]?subject=unsubscribe
