Re: [Wireshark-dev] [pcap-ng-format] Proposal for storing decryption secrets in a pcapng block
On Oct 5, 2018, at 6:47 AM, Michael Richardson wrote:
> Guy Harris wrote:
>> The second and third option require either the producer, or some
>> post-processor, to write a new version of the file putting the secrets
>> before the packets that require them. The producer isn't necessarily
>> responsible for doing so; one might have tcpdump, or dumpcap (or some
>> program using dumpcap, such as TShark or Wireshark) write out a capture
>> with no secrets, and then have another program (a utility, or Wireshark
>> after having read in the file and then given the secret in question)
>> write out a new file with the secrets early enough in the file ("before
>> all the packet blocks" is probably the simplest implementation).
>
> I'm in favour of this option, and providing a signal early in the file that
> the indicates if that process has occured yet.
"That process" being the process of adding all relevant secrets to the file?
For what would that indication be used?
___
Sent via:Wireshark-dev mailing list
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] [pcap-ng-format] Proposal for storing decryption secrets in a pcapng block
On Oct 6, 2018, at 2:24 AM, Anders Broman wrote: >> The newline requirement is specified in the current proposal >> https://github.com/pcapng/pcapng/pull/54/files#diff-83e833afc268a7be6b36d2b897656e28R1912 >> >> There is no explicit text on concatenation though. > > This presumes the secret block is a text block rather than a binary blob? It presumes that the TLS session secrets blob is text rather than binary; it doesn't require that *all* secrets be text - the description of the Secrets field is just "binary octets containing secrets". ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] [pcap-ng-format] Proposal for storing decryption secrets in a pcapng block
What Guy said.
On Fri, Oct 5, 2018 at 4:11 PM Guy Harris wrote:
>
> On Sep 30, 2018, at 10:47 AM, Peter Wu wrote:
>
> > Requirements for block placement:
> > - No requirement. Producers are allowed to write the block anywhere.
> > Disadvantages for consumers: requires a two-pass scan to collect
> > secrets before they are used.
> > - Place secrets before the packet blocks that require them. Consumers
> > can read and decrypt in one pass. Disadvantage: producers cannot
> > always guarantee availability of secrets while writing the capture.
> > - Place a single secret block before the first packet block. Consumers
> > can read and decrypt in one pass. Disadvantage: requires producers to
> > post-process (rewrite) the capture file to insert secrets.
>
> The third of those appears to be a special case of the second of those. I
> don't see any need to require the secrets to be before the *first* packet
> block if the first packet block doesn't require the secret; presumably
> "before the packet blocks that require them" just means "*somewhere* before
> the packet blocks that require them", which is *allowed* to be "before all
> packet blocks in the file" but not *required* to be "before all packet blocks
> in the file".
>
> If the secret isn't available by the time the first packet requiring the
> secret for decryption is ready to be written to the capture, *somebody* will
> have to do some form of two-pass processing.
>
> The first option says the consumer must do so; that's inconvenient for a
> consumer doing one-pass processing (tcpdump, TShark without the -2 option),
> and isn't even really good for at least some consumers doing two-pass
> processing (Wireshark, TShark with the -2 option), because dissection is done
> on the first pass.
>
> The second and third option require either the producer, or some
> post-processor, to write a new version of the file putting the secrets before
> the packets that require them. The producer isn't necessarily responsible
> for doing so; one might have tcpdump, or dumpcap (or some program using
> dumpcap, such as TShark or Wireshark) write out a capture with no secrets,
> and then have another program (a utility, or Wireshark after having read in
> the file and then given the secret in question) write out a new file with the
> secrets early enough in the file ("before all the packet blocks" is probably
> the simplest implementation).
>
> A producer that *does* happen to have the secret available before seeing any
> packets that require the secret *could* write it directly.
> ___
> Sent via:Wireshark-dev mailing list
> Archives:https://www.wireshark.org/lists/wireshark-dev
> Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
> mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
___
Sent via:Wireshark-dev mailing list
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] [pcap-ng-format] Proposal for storing decryption secrets in a pcapng block
Den lör 6 okt. 2018 10:51Peter Wu skrev:
> On Thu, Oct 04, 2018 at 03:12:19PM -0700, Ben Higgins wrote:
> > On Sun, Sep 30, 2018 at 10:47 AM Peter Wu wrote:
> >
> > > Hi all,
> > >
> > > Earlier this year, Ben Higgins proposed a new pcapng block to store
> > > SSL/TLS session secrets that would allow users to enable decryption of
> > > packet traces without further configuration. I would like to solicit
> for
> > > some feedback on this proposed specification update:
> > > https://github.com/pcapng/pcapng/pull/54
> > >
> > > Among the open spec issues:
> > > - Are you happy with the chosen identifiers (10 for block type and
> > > 0x544c534b ("TLSK") for the TLS key log secret type).
> > > - Rename the block from the original proposal (it seems based on "IDB",
> > > but "Decryption Secrets Block (DSB)" sounds better to me).
> > >
> >
> > Both these sound good to me.
> >
> > - Is there a use case for multiple secret blocks?
> > >
> >
> > Certainly if you have different secret block types (so you might need one
> > of each).
>
> Anders suggested having a TLV (which is present in the current
> proposal). If that was hypothetically extended with support for multiple
> TLVs, then multiple secrets could be stored in a single DSB. Though that
> brings more complexity for the consumer and might not be worth it.
>
> > Even for the same type it'd make it easier on producers that
> > might not know the length of all secrets up front (i.e. it's filling up a
> > buffer as it goes and spitting out a secret block once the buffer's
> full).
>
> This is indeed a valid concern, and is a reason why the current spec
> text allows multiple blocks.
>
> > - For multiple secret blocks, is concatenation a good merge strategy?
> > >
> >
> > Concatenation should work fine among TLSK blocks assuming all blocks
> have a
> > final newline (or one is inserted if missing during concat; perhaps that
> > needs to be specified).
>
> The newline requirement is specified in the current proposal
>
> https://github.com/pcapng/pcapng/pull/54/files#diff-83e833afc268a7be6b36d2b897656e28R1912
>
> There is no explicit text on concatenation though.
>
This presumes the secret block is a text block rather than a binary blob?
> --
> Kind regards,
> Peter Wu
> https://lekensteyn.nl
> ___
> Sent via:Wireshark-dev mailing list
> Archives:https://www.wireshark.org/lists/wireshark-dev
> Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
> mailto:wireshark-dev-requ...@wireshark.org
> ?subject=unsubscribe
___
Sent via:Wireshark-dev mailing list
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] [pcap-ng-format] Proposal for storing decryption secrets in a pcapng block
On Thu, Oct 04, 2018 at 03:12:19PM -0700, Ben Higgins wrote:
> On Sun, Sep 30, 2018 at 10:47 AM Peter Wu wrote:
>
> > Hi all,
> >
> > Earlier this year, Ben Higgins proposed a new pcapng block to store
> > SSL/TLS session secrets that would allow users to enable decryption of
> > packet traces without further configuration. I would like to solicit for
> > some feedback on this proposed specification update:
> > https://github.com/pcapng/pcapng/pull/54
> >
> > Among the open spec issues:
> > - Are you happy with the chosen identifiers (10 for block type and
> > 0x544c534b ("TLSK") for the TLS key log secret type).
> > - Rename the block from the original proposal (it seems based on "IDB",
> > but "Decryption Secrets Block (DSB)" sounds better to me).
> >
>
> Both these sound good to me.
>
> - Is there a use case for multiple secret blocks?
> >
>
> Certainly if you have different secret block types (so you might need one
> of each).
Anders suggested having a TLV (which is present in the current
proposal). If that was hypothetically extended with support for multiple
TLVs, then multiple secrets could be stored in a single DSB. Though that
brings more complexity for the consumer and might not be worth it.
> Even for the same type it'd make it easier on producers that
> might not know the length of all secrets up front (i.e. it's filling up a
> buffer as it goes and spitting out a secret block once the buffer's full).
This is indeed a valid concern, and is a reason why the current spec
text allows multiple blocks.
> - For multiple secret blocks, is concatenation a good merge strategy?
> >
>
> Concatenation should work fine among TLSK blocks assuming all blocks have a
> final newline (or one is inserted if missing during concat; perhaps that
> needs to be specified).
The newline requirement is specified in the current proposal
https://github.com/pcapng/pcapng/pull/54/files#diff-83e833afc268a7be6b36d2b897656e28R1912
There is no explicit text on concatenation though.
--
Kind regards,
Peter Wu
https://lekensteyn.nl
___
Sent via:Wireshark-dev mailing list
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] [pcap-ng-format] Proposal for storing decryption secrets in a pcapng block
Guy Harris wrote:
> The second and third option require either the producer, or some
> post-processor, to write a new version of the file putting the secrets
> before the packets that require them. The producer isn't necessarily
> responsible for doing so; one might have tcpdump, or dumpcap (or some
> program using dumpcap, such as TShark or Wireshark) write out a capture
> with no secrets, and then have another program (a utility, or Wireshark
> after having read in the file and then given the secret in question)
> write out a new file with the secrets early enough in the file ("before
> all the packet blocks" is probably the simplest implementation).
I'm in favour of this option, and providing a signal early in the file that
the indicates if that process has occured yet.
> A producer that *does* happen to have the secret available before
> seeing any packets that require the secret *could* write it directly.
Agreed.
--
] Never tell me the odds! | ipv6 mesh networks [
] Michael Richardson, Sandelman Software Works| network architect [
] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[
___
Sent via:Wireshark-dev mailing list
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] [pcap-ng-format] Proposal for storing decryption secrets in a pcapng block
On Sep 30, 2018, at 10:47 AM, Peter Wu wrote:
> Requirements for block placement:
> - No requirement. Producers are allowed to write the block anywhere.
> Disadvantages for consumers: requires a two-pass scan to collect
> secrets before they are used.
> - Place secrets before the packet blocks that require them. Consumers
> can read and decrypt in one pass. Disadvantage: producers cannot
> always guarantee availability of secrets while writing the capture.
> - Place a single secret block before the first packet block. Consumers
> can read and decrypt in one pass. Disadvantage: requires producers to
> post-process (rewrite) the capture file to insert secrets.
The third of those appears to be a special case of the second of those. I
don't see any need to require the secrets to be before the *first* packet block
if the first packet block doesn't require the secret; presumably "before the
packet blocks that require them" just means "*somewhere* before the packet
blocks that require them", which is *allowed* to be "before all packet blocks
in the file" but not *required* to be "before all packet blocks in the file".
If the secret isn't available by the time the first packet requiring the secret
for decryption is ready to be written to the capture, *somebody* will have to
do some form of two-pass processing.
The first option says the consumer must do so; that's inconvenient for a
consumer doing one-pass processing (tcpdump, TShark without the -2 option), and
isn't even really good for at least some consumers doing two-pass processing
(Wireshark, TShark with the -2 option), because dissection is done on the first
pass.
The second and third option require either the producer, or some
post-processor, to write a new version of the file putting the secrets before
the packets that require them. The producer isn't necessarily responsible for
doing so; one might have tcpdump, or dumpcap (or some program using dumpcap,
such as TShark or Wireshark) write out a capture with no secrets, and then have
another program (a utility, or Wireshark after having read in the file and then
given the secret in question) write out a new file with the secrets early
enough in the file ("before all the packet blocks" is probably the simplest
implementation).
A producer that *does* happen to have the secret available before seeing any
packets that require the secret *could* write it directly.
___
Sent via:Wireshark-dev mailing list
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] [pcap-ng-format] Proposal for storing decryption secrets in a pcapng block
On Sun, Sep 30, 2018 at 10:47 AM Peter Wu wrote:
> Hi all,
>
> Earlier this year, Ben Higgins proposed a new pcapng block to store
> SSL/TLS session secrets that would allow users to enable decryption of
> packet traces without further configuration. I would like to solicit for
> some feedback on this proposed specification update:
> https://github.com/pcapng/pcapng/pull/54
>
> Among the open spec issues:
> - Are you happy with the chosen identifiers (10 for block type and
> 0x544c534b ("TLSK") for the TLS key log secret type).
> - Rename the block from the original proposal (it seems based on "IDB",
> but "Decryption Secrets Block (DSB)" sounds better to me).
>
Both these sound good to me.
- Is there a use case for multiple secret blocks?
>
Certainly if you have different secret block types (so you might need one
of each). Even for the same type it'd make it easier on producers that
might not know the length of all secrets up front (i.e. it's filling up a
buffer as it goes and spitting out a secret block once the buffer's full).
- For multiple secret blocks, is concatenation a good merge strategy?
>
Concatenation should work fine among TLSK blocks assuming all blocks have a
final newline (or one is inserted if missing during concat; perhaps that
needs to be specified).
- Is this format future-proof and usable for other formats like ZigBee?
>
Not sure if the merge strategy could be uniform among other secret formats,
but otherwise this spec seems future-proof since a new secret type can
entail a new secret format.
> Advantages of allowing multiple blocks:
> - Producers can write secrets directly while writing packets.
> - Merging multiple capture files is simpler.
>
> Requirements for block placement:
> - No requirement. Producers are allowed to write the block anywhere.
> Disadvantages for consumers: requires a two-pass scan to collect
> secrets before they are used.
> - Place secrets before the packet blocks that require them. Consumers
> can read and decrypt in one pass. Disadvantage: producers cannot
> always guarantee availability of secrets while writing the capture.
>
- Place a single secret block before the first packet block. Consumers
> can read and decrypt in one pass. Disadvantage: requires producers to
> post-process (rewrite) the capture file to insert secrets.
>
I'm fine with the second option given that, as you note, "No requirement"
is more challenging on consumers.
As these blocks contain sensitive (session) secrets, they should be
> carefully handled, but that's probably a different discussion. The
> current Wireshark patches that implement *read-only* support is at
> https://code.wireshark.org/review/29901
>
> Your feedback is welcome.
> --
> Kind regards,
> Peter Wu
> https://lekensteyn.nl
> ___
> pcap-ng-format mailing list
> pcap-ng-for...@winpcap.org
> https://www.winpcap.org/mailman/listinfo/pcap-ng-format
___
Sent via:Wireshark-dev mailing list
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] [pcap-ng-format] Proposal for storing decryption secrets in a pcapng block
Michael Richardson wrote: > Peter Wu wrote: > > Requirements for block placement: > > - No requirement. Producers are allowed to write the block anywhere. > > Disadvantages for consumers: requires a two-pass scan to collect > > secrets before they are used. > I prefer this, but I would support having a flag in the block that says that > no other blocks exist in the file until at least X-bytes. > So, a producer (or something downstream of it), could scan for the blocks, > move them to the front, and indicate how far into the file it cover. > Naturally, if X >= file size, then the work is done. I agree that this would be nice but I see technical difficulties with this. When writing a block you have to assume that you don't know what's going to be written next, so you don't know how far it is to the next block. pcap-ng files are usually written by the producer as a stream of blocks, so you can't go back to update a previous block when you write the next one. Also, moving blocks around while writing a live capture is not an option when it comes to heavy loads. Or did I misunderstand your idea? ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
Re: [Wireshark-dev] [pcap-ng-format] Proposal for storing decryption secrets in a pcapng block
Peter Wu wrote: > Requirements for block placement: > - No requirement. Producers are allowed to write the block anywhere. > Disadvantages for consumers: requires a two-pass scan to collect > secrets before they are used. I prefer this, but I would support having a flag in the block that says that no other blocks exist in the file until at least X-bytes. So, a producer (or something downstream of it), could scan for the blocks, move them to the front, and indicate how far into the file it cover. Naturally, if X >= file size, then the work is done. > - Place secrets before the packet blocks that require them. Consumers > can read and decrypt in one pass. Disadvantage: producers cannot > always guarantee availability of secrets while writing the capture. > - Place a single secret block before the first packet block. Consumers > can read and decrypt in one pass. Disadvantage: requires producers to > post-process (rewrite) the capture file to insert secrets. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works| network architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ ___ Sent via:Wireshark-dev mailing list Archives:https://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
