[
https://issues.apache.org/jira/browse/DISPATCH-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078200#comment-16078200
]
ASF GitHub Bot commented on DISPATCH-760:
-----------------------------------------
GitHub user ChugR opened a pull request:
https://github.com/apache/qpid-dispatch/pull/173
DISPATCH-760: Message annotation handling improvements
This PR replaces existing annotation handling with a more efficient
version. It is fully compatible with 0.8.x.
It contains the same stuff as
[pr171](https://github.com/apache/qpid-dispatch/pull/171) with the addition of
five commits (b321fa2..50637b7) to clean up and satisfy review comments
A comparison with 0.8.x has run on a single laptop. A sender sends a short
message with a varying number of 10-byte user annotations to an ingress router
A. Router A sends the messages to egress router B. A single client is attached
to router B. This comparison is not rigorous and at the
50-annotation-per-message level the producer started to be the limiting factor.
Messages per second. Number of annotations, Nanno, vs router code version,
0.8.x or new.
| Nanno | 0.8.x | new |
|----------|-----------|--------|
| 0 | 26500 | 32000 |
| 1 | 24000 | 31200 |
| 5 | 18000 | 30100 |
| 10 | 14200 | 28500 |
| 20 | 8200 | 24400 |
| 50 | 3500 | 11300 |
This PR performance numbers are down a little bit from the last PR. This PR
does a little extra work to prevent router annotations from being spoofed by
user annotations.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ChugR/qpid-dispatch crolke-DISPATCH-760-3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/qpid-dispatch/pull/173.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #173
----
commit 1c3b0dd6f2e64718242c3da152ae56312ab2d365
Author: Chuck Rolke <[email protected]>
Date: 2017-06-30T15:04:17Z
DISPATCH-760: accessor for connection status: interrouter vs user
commit 6e8b56e5b5b7b2a5ea1e65b34c2f93c885bbb599
Author: Chuck Rolke <[email protected]>
Date: 2017-06-30T15:46:43Z
DISPATCH-760: add opaque compose map item insertion functions
commit cf2c76a924de2a487a92d294b2f2de39506a30ee
Author: Chuck Rolke <[email protected]>
Date: 2017-06-30T15:56:08Z
DISPATCH-760: promote pointer_t to public qd_iterator_pointer_t
commit 7f650d8f9f3106942201b0b6ddc7eb4fe434a9a9
Author: Chuck Rolke <[email protected]>
Date: 2017-06-30T16:30:35Z
DISPATCH-760: Add prefix search functions that use iterator pointers
commit b67b2013fd6c666c80119704821b22c8ee431257
Author: Chuck Rolke <[email protected]>
Date: 2017-06-30T20:57:17Z
DISPATCH-760: Add new annotation parser
commit 21014c6e9f8dd2ea05551f341487a9d486c074cd
Author: Chuck Rolke <[email protected]>
Date: 2017-07-01T19:01:56Z
DISPATCH-760: Message handling uses new scheme
commit 3b05873fcd0f04fda3682e42b024be0ff60babca
Author: Chuck Rolke <[email protected]>
Date: 2017-07-01T19:51:00Z
DISPATCH-760: compare key names using strcmp not iter_equal
commit 95a05563a71cba7a46ce5f462a89e4f6b0e610d7
Author: Chuck Rolke <[email protected]>
Date: 2017-07-03T15:23:39Z
DISPATCH-760: Get self tests going
Get rid of is_interrouter flag in favor of existing strip_annotations flags.
Pass strip_annotations_in through message to parse function.
commit cd7d5128ea9ece90ce1236ad027f5c4f448656df
Author: Chuck Rolke <[email protected]>
Date: 2017-07-03T16:05:36Z
DISPATCH-760: Fix inbound annotation stripping
commit b321fa26579470107fbeadc3815488155db273fd
Author: Chuck Rolke <[email protected]>
Date: 2017-07-05T21:45:06Z
DISPATCH-760: don't re-strip inbound annotations
commit 40cb7c1ef8811b1a948f9233cd99e128d3089c15
Author: Chuck Rolke <[email protected]>
Date: 2017-07-06T11:03:16Z
DISPATCH-760: improve annotation V0 logic
commit 425c898f623b973d30e98e1e971c70faff944da0
Author: Chuck Rolke <[email protected]>
Date: 2017-07-06T13:03:41Z
DISPATCH-760: Prevent user anno's from being considered in routing logic
Always provide four routing annotations by padding if necessary.
commit 9672b71912f50a18df0ce392e8775077336b5553
Author: Chuck Rolke <[email protected]>
Date: 2017-07-06T17:16:03Z
DISPATCH-760: remove strip flag
The flag is only set false and is not needed.
commit 50637b770400e106bfb1c2e2ad7c57f8f8b2321e
Author: Chuck Rolke <[email protected]>
Date: 2017-07-06T20:14:35Z
DISPATCH-760: Change magic numbers into formal constants
Router message annotation uses some of map keys and will at most
use QD_MA_N_KEYS annotation entries.
The longest key strlen is QD_MA_MAX_KEY_LEN bytes.
Router message annotation keys are at the end of the annotation
section. Code uses a sliding window to avoid searching each key
for the router prefix that indicates a router annotation. The
size of the sliding filter is QD_MA_FILTER_LEN map entries.
When QD_MA_FILTER_LEN is larger than QD_MA_N_KEYS then the extra
space may hold a future router version's new keys and successfully
strip them from the input stream.
----
> Inefficiencies handling message annotations
> -------------------------------------------
>
> Key: DISPATCH-760
> URL: https://issues.apache.org/jira/browse/DISPATCH-760
> Project: Qpid Dispatch
> Issue Type: Bug
> Affects Versions: 0.8.0
> Environment: Fedora 25 laptop
> Single router, 8 worker threads, single receiver, single sender. Messages
> have 200 100-byte annotations.
> Reporter: Chuck Rolke
> Assignee: Chuck Rolke
> Attachments: DISPATCH-760-ALL-WORKERS.svg,
> DISPATCH-760-core-thread.svg
>
>
> The router as a whole spends a lot of time managing annotations. With routers
> and other intermediaries adding and stripping annotations, improvements in
> this area could be valuable.
> Testing a stand-alone router gets these results.
> ||N 100-byte annotations || msg/S throughput||
> | 0 | 30,000 |
> | 1 | 22,500 |
> | 2 | 20,500 |
> | 5 | 15,500 |
> | 10 | 10,000 |
> | 20 | 5,500 |
> | 50 | 2,400 |
> | 100 | 1,200 |
> | 200 | 500 |
> Application 'perf' provides some visibility into where the threads are
> spending their time. View the attached .svg files to see what the core thread
> and an aggregation of the remaining threads are busy doing.
> The worker threads are spending ~85% of their time composing messages while
> the core thread is spending 90+% of its time deleting deliveries with a
> majority of its time in qd_parse_free().
> A 'top' thread snapshot shows several worker threads completely idle. The
> core thread has tid=32631.
> {noformat}
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 32633 chug 20 0 858376 137708 8484 R 26.7 0.9 0:33.99 qdrouterd
> 32632 chug 20 0 858376 137708 8484 S 20.0 0.9 0:34.63 qdrouterd
> 32631 chug 20 0 858376 137708 8484 S 13.3 0.9 0:30.13 qdrouterd
> 32635 chug 20 0 858376 137708 8484 R 13.3 0.9 0:34.26 qdrouterd
> 32636 chug 20 0 858376 137708 8484 S 13.3 0.9 0:34.80 qdrouterd
> 32638 chug 20 0 858376 137708 8484 S 13.3 0.9 0:34.34 qdrouterd
> 32616 chug 20 0 858376 137708 8484 S 6.7 0.9 0:35.48 qdrouterd
> 32634 chug 20 0 858376 137708 8484 S 6.7 0.9 0:34.09 qdrouterd
> 32637 chug 20 0 858376 137708 8484 S 6.7 0.9 0:34.42 qdrouterd
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
