I thought of a potential performance improvement that you might consider.
The PCAP schema was really written with the notion of educating people on how to use dfdl:inputValueCalc, dfdl:outputValueCalc, to show how to deal with byte order that is determined at runtime, etc. If you can do without most of the computed elements, and perhaps the runtime byte order, then I think things could speed up quite a bit. The PCAP schema contains a bunch of computed-fields that are there to make the IP addresses show up in the parsed XML in the usual 10.10.10.10 dot-separated notation. But if you think about it, this syntax for an IP address is not actually parsed. E.g., if <IPAddress><a>10</a><a>10</a><a>10</a><a>10</a></IPAddress> - which is what a real XML parse of an IP address naturally looks like, if that is ok for you or even one big 32-bit integer or hexBinary "0A0A0A0A" versus having to have a non-XML notation like "10.10.10.10", then a bunch of stuff goes away that is quite expensive relative to just ordinary elements. Also anything you can share with this mailing list w.r.t. your performance requirements would be helpful for planning purposes. -mike beckerle ________________________________ From: Mike Beckerle Sent: Wednesday, May 2, 2018 8:50:26 AM To: [email protected] Cc: Clarke, Frederick Subject: Re: Daffodil tuning Weber, Is this for PCAP data? or some other schema? We would need to look in detail at the schema unless it's one we already know. I doubt there are any magic tuning knobs that will massively speed things up. Daffodil is largely compute bound, and while we've improved the performance a bit, there is much more work to do here, especially on unparsing, which for formats like PCAP where values must be calculated to be stored into headers, those mechanisms which suspend unparsing, double buffer data, until the needed length value can be computed, and then resume unparsing once it is known,... those mechansims have some real overhead. Expressions are being evaluated, suspended, retried, etc. This page on the Apache Daffodil Wiki https://cwiki.apache.org/confluence/display/DAFFODIL/Unparsing%3A+Data+Output+Stream+Buffering+and+dfdl%3AoutputValueCalc may give you some idea of the complexity involved in unparsing. Are your tests both parsing and unparsing the data? I suspect unparsing is the bottleneck, but it would be good to tease those apart if possible. --Mike Beckerle Tresys ________________________________ From: Perpignand, Weber <[email protected]> Sent: Tuesday, May 1, 2018 4:43:44 PM To: [email protected] Cc: Clarke, Frederick Subject: Daffodil tuning Hello, I have been running some performance tests on incubator-daffodil-master. The best throughput that I have been able to get is 100 Mb/sec on a full frame (1518 bytes) packet. I changed the java -Xmx1024m to -Xmx2048m and the cache size from 128 to 256, but this did not give better result. Are there other tuning parameters you can recommend? I do see the tunable options in daffodil (DaffodilTunables.scala), but these also had no noticeable effect. Thanks. Weber Perpignand Senior Software Engineer +1 (203) 894-9342 x189 [email protected]<mailto:[email protected]> www.OwlCyberDefense.com<http://www.owlcyberdefense.com/> [Email Signature Logo] See us at the following 2018 events: Rome Labs/Griffiss Park, Rome, NY (May 10th)<https://www.owlcyberdefense.com/events> DOT Cybersecurity Summit, Washington, D.C. (May 23rd – May 24th)<https://www.owlcyberdefense.com/events> Defense Industry Forum, Warsaw, Poland - Speaking! (May 23rd)<https://www.owlcyberdefense.com/events>
