On 7/8/22 20:02, Adrian Moreno wrote:
> While troubleshooting or developing new features in OVS, a considerable
> amount of time is spent analyzing flows (whether that's Openflow flows
> or datapath flows). Currently, OVS has tools to dump flows with
> different levels of verbosity as well as to filter flows prior to
> dumping them, e.g: 'ovs-ofctl dump-flows', 'ovs-appctl
> dpctl/dump-flows', etc.
>
> The output of these commands is considered stable so it should be
> possible to write more layered tools that enable advanced flow analysis.
> However, the way flows are formatted into strings is not trivial to
> parse.
>
> This series introduces the a flow parsing library capable of parsing
> both Openflow and DPIF flows.
>
> The library is based on generic key-value and list parsers and a number
> of common decoders. Based on that, an Openflow flow parser and a DPIF
> flow parser are introduced by defining a way to decode each possible
> field and action they might contain.
>
> The library has the following features:
> - Parsed key-value pairs keep some metadata that refer to the original
> strings they were extracted from. That way the flows can be printed
> and formatted in flexible ways.
> - It includes a basic flow filtering mechanism. A filter can be defined
> combining logical (||, &&, !), arithmetical (<, >, =) or mask (~=)
> operations
> - It supports IPAddress and Ethernet masking (based on netaddr)
> - The decoder to use for each key (match or action) is set explicitly to
> avoid expensive runtime type-guessing.
> - The decoders to use for Openflow fields is automatically generated
> based on meta-flow.h
> - Additional dependencies:
> - netaddr: For IP and Ethernet Address management.
> - pyparsing: For filtering syntax.
> - pytest: For unit tests.
>
> One key goal challenge of including this library is avoiding diversion
> between the C code that prints/parses the flows and the python parsing
> code. To that effect, the series introduces the following mechanisms:
> - Decoding information of openflow fields is automatically generated
> based on meta-flow.h
> - The calls to ovs-ofctl made from tests/ofp-actions.at are wrapped by a
> python script that also runs the python parsers. If an exception is
> raised by the python code (meaning it was not capable of parsing the
> flow string), the test will fail
> - The calls to the test-odp made from tests/odp.at are wrapped by a
> python script that also runs the python parsers. If an exception is
> raised by the python code (meaning it was not capable of parsing the
> flow string), the test will fail.
> - A python unit test testsuite ensures python code works and it's easy
> to add more flow examples to it
> - A dependency check is introduced. The python parsing code mainly
> depends on lib/ofp-actions.c and lib/odp-util.c. This series stores
> the md5sum of such files and adds a build target that ensures the
> files have not been changed. That way, anyone who modifies those files
> will get a warning the fist time they build the project. Dependency
> digests are easily updated using a string so hopefully this warning
> would not be too inconvenient.
>
> Library usage
> -------------
>>>> from ovs.flows.ofp import OFPFlow
>>>> flow = OFPFlow("cookie=0x2b32ab4d, table=41, n_packets=11, n_bytes=462,
>>>> priority=33,ip,reg15=0x2/0x2,nw_src=10.128.0.2/24
>>>> actions=move:NXM_OF_TCP_DST[]->NXM_NX_XXREG0[32..47],ct(table=16,zone=NXM_NX_REG13[0..15],nat)")
>>>> flow.info
> {'cookie': 724740941, 'table': 41, 'n_packets': 11, 'n_bytes': 462}
>>>> flow.match
> {'priority': 33, 'ip': True, 'reg15': Mask32('0x2/0x2'), 'nw_src':
> IPMask('10.128.0.2/24')}
>>>> flow.actions
> [{'move': {'src': {'field': 'NXM_OF_TCP_DST'}, 'dst': {'field':
> 'NXM_NX_XXREG0', 'start': 32, 'end': 47}}}, {'ct': {'table': 16, 'zone':
> {'field': 'NXM_NX_REG13', 'start': 0, 'end': 15}, 'nat': True}}]
>>>> from ovs.flows.filter import OFFilter
>>>> filt = OFFilter("nw_src ~= 10.128.0.10 and (table = 42 or n_packets > 0)")
>>>> filt.evaluate(flow)
> True
>
> V4 -> V5:
> - rename tests/test-ovs-{dp,of}parse.py to tests/test-{dp,of}parse.py
> - add dependencies to netaddr and pyparsing.
> In pip: add a new extras_requirement called "flow"
> In packages: add them in the Suggests section
> In CI: install them in linux-prepare.sh
>
> V3 -> V4:
> - renamed the library ovs/flows -> ovs/flow. Note I've kept Acked-by
> tags for patches where only this change was done.
> - unit tests: instead of wrapping ovs-ofctl/test-odp tools, just add
> another test to the correspondent *.at files as per Ilya's suggestion.
> - Dropped patch 13.
> - Addressed Ilya's comments on unit test checks (old patch 14/18, new
> patch 13/17).
>
> V2 -> V3:
> - Simplified KV and list decoding code (Mark Michelson's suggestion)
> - Fixed typos
> - Added missing files to FLAKE8_PYFILES
> - Go back to a simplified ipv4/6 regexp for ip-port range extraction.
> Also added specific unit test for ip-port range decoding.
> - Adapt ofp encap() action decoder to support new header types: mpls and
> mpls_mc
> (the need for change was detected by patch 13)
>
> V1 -> V2:
> - list/kv parsers: changed the API to accept the string to parse in the
> constructor.
> - list/kv parsers: allow re.split() to return less than 3 elements
> (enables support for python 3.6).
> - decoders: add a more accurate IPv6 regexp and remove confusing max()
> in IPMask and EthMask.
> - odp/ofp flows: remove the *Factory class and implement caching of
> decoders using class variables.
> - odp/ofp flows: homogenize names of functions and made them static.
> - moved pytest unit tests from a build target to a testsuite and added
> their requirements to python/test_requirements.txt.
> - Formatting fixes and missing dots (lots of them!).
>
> RFC -> V1:
> - filters: created a class to represent the filtering result. That way
> we can store more information such as what key actually triggered the
> match. This enables functionality such as highlighting of keys based
> on an expression
> - Formatted python code according to flake8 requirements
> - Split ofp actions in ofp_act.py
> - drop ofparse utility (will send a RFC to the mail list soon)
> - Moved the initialization of the decoders objects to a factory so they
> are cached. This significantly decreases parsing time of large dumps.
>
>
> Adrian Moreno (17):
> python: add generic Key-Value parser
> python: add mask, ip and eth decoders
> python: add list parser
> build-aux: split extract-ofp-fields
> build-aux: generate ofp field decoders
> python: add flow base class
> python: introduce OpenFlow Flow parsing
> python: add ovs datapath flow parsing
> python: add flow filtering syntax
> python: add a json encoder to flow fields
> tests: verify flows in ofp-actions are parseable
> tests: verify flows in odp.at are parseable
> python: introduce unit tests
> python: add unit tests for ListParser
> python: add unit tests for openflow parsing
> python: add unit tests to datapath parsing
> python: add unit tests for filtering engine
Thanks, folks! That's a nice library to have and the unit test
framework can be re-used for testing other python parts in the
future. I held this patch set back for a while since I knew it
conflicts with the packaging update. I adjusted it and applied
now. Mostly.
I dropped 2 patches with tests-(od,dp)parse.py, because they are
not actually testing anything. The flow parsing library is unable
to fail due to default decoders that just resolve everything they
do not understand into strings. And to_string() method returns
the original string, not what was parsed. So, both test utilities
are not helpful. Since it's a nice to have testing and not
necessary one (the library is still tested by pytest), I decided
to just drop these 2 patches from the set and apply the rest.
Adrian, it would be great if you can re-work the dropped patches
though, since they are supposed to be helpful. I thought of maybe
adding support for 'no_defaults' configuration, so the library
will fail on any unknown field. Is that possible?
Best regards, Ilya Maximets.
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
