[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491240#comment-16491240
]
ASF subversion and git services commented on QPID-7830:
---
Commit 57bda83de7fbe6291aea507b5c1098fda5cf130d in qpid-broker-j's branch
refs/heads/master from [~alex.rufous]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=57bda83 ]
QPID-7830: [Broker-J] Simplify string caching functionality
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Assignee: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4
>
> Attachments:
> 0001-QPID-7830-Broker-J-Simplify-string-caching-functiona.patch
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489319#comment-16489319
]
ASF subversion and git services commented on QPID-7830:
---
Commit 28e73f3c1abad32d69d4e2f0937f1021a3ac61dd in qpid-broker-j's branch
refs/heads/7.0.x from [~alex.rufous]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=28e73f3 ]
QPID-7830: [Broker-J] Fix failing test
(cherry picked from commit 196b27065219b1283522bdb3a8f4fb1dda82ac16)
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Assignee: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4
>
> Attachments:
> 0001-QPID-7830-Broker-J-Simplify-string-caching-functiona.patch
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489321#comment-16489321
]
ASF subversion and git services commented on QPID-7830:
---
Commit 63f7b7e949ac04ee7596ef6b78b8160ab146 in qpid-broker-j's branch
refs/heads/7.0.x from [~alex.rufous]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=63f7b7e ]
QPID-7830: [Broker-J] Add null checks for 'subject' and 'to'
(cherry picked from commit dc2e9d425a4547e0efbf6d8de70ec0c90ecdbdc4)
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Assignee: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4
>
> Attachments:
> 0001-QPID-7830-Broker-J-Simplify-string-caching-functiona.patch
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489320#comment-16489320
]
ASF subversion and git services commented on QPID-7830:
---
Commit c14eb33b5062b81e9475215fc0f90b21026b4eae in qpid-broker-j's branch
refs/heads/7.0.x from [~alex.rufous]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=c14eb33 ]
QPID-7830: [Broker-J] Cache AMQP 1.0 application property names and message
properties 'to' and 'subject'
(cherry picked from commit 85f1db6c4a2b5127ebec78cedcbe4563094cab47)
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Assignee: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4
>
> Attachments:
> 0001-QPID-7830-Broker-J-Simplify-string-caching-functiona.patch
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489318#comment-16489318
]
ASF subversion and git services commented on QPID-7830:
---
Commit 7b675cb7359237a7023332f42471563c043fc4e5 in qpid-broker-j's branch
refs/heads/7.0.x from [~alex.rufous]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=7b675cb ]
QPID-7830: [Broker-J] Cache encoded and decoded values for 0-10 str8
(cherry picked from commit 5b6dfcd94ef64544040de53d38e2463d327ebcb1)
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Assignee: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4
>
> Attachments:
> 0001-QPID-7830-Broker-J-Simplify-string-caching-functiona.patch
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489306#comment-16489306
]
ASF subversion and git services commented on QPID-7830:
---
Commit dc2e9d425a4547e0efbf6d8de70ec0c90ecdbdc4 in qpid-broker-j's branch
refs/heads/master from [~alex.rufous]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=dc2e9d4 ]
QPID-7830: [Broker-J] Add null checks for 'subject' and 'to'
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Assignee: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4
>
> Attachments:
> 0001-QPID-7830-Broker-J-Simplify-string-caching-functiona.patch
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488982#comment-16488982
]
Alex Rudyy commented on QPID-7830:
--
The caching of strings in AMQP 0-8 and AMQP 1.0 is implemented only for
certain fields. It looks like inconsistent approach to me. I think that caching
should be implemented in the layer decoding the string regardless the string
use case. Caching of string in AMQP 0-10 is implemented in decoder and looks to
me like a more attractive approach than the one used for caching in AMQP 0-8
and AMQP 1.0. Though it is less effective as single-use object would hit the
cache but it is easier to maintain and reason about. Currently, in AMQP 0-8 we
do not cache message header names. Potentially caching of header names in AMQP
0-8 could further reduce memory consumption.
I believe that caching functionality should be moved into
{{StringTypeConstructor}} in AMQP 1.0 and implemented as part of {{construct}},
something like below
{code}
byte[] data = new byte[size];
in.get(data);
ByteBuffer buffer = ByteBuffer.wrap(data);
String cached = CACHE.get().getIfPresent(buffer);
if (cached == null)
{
cached = new String(data, UTF_8);
CACHE.get().put(buffer, cached);
}
return cached;
{code}
AMQP 0-8 {{AMQPShortStrings}} should be cached unconditionally in
{{AMQShortString#readAMQShortString}} similar to code above and calls to
{{intern}} should be removed. That should simplify the code and would allow to
cache more duplicate string values than the current approach.
I am going to attach a patch for the suggested changes. In meant time, I am
going to merge implemented changes into 7.0.x
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Assignee: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.1.0, qpid-java-broker-7.0.4
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488919#comment-16488919
]
ASF subversion and git services commented on QPID-7830:
---
Commit 85f1db6c4a2b5127ebec78cedcbe4563094cab47 in qpid-broker-j's branch
refs/heads/master from [~alex.rufous]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=85f1db6 ]
QPID-7830: [Broker-J] Cache AMQP 1.0 application property names and message
properties 'to' and 'subject'
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Priority: Major
> Fix For: Future
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472218#comment-16472218
]
ASF subversion and git services commented on QPID-7830:
---
Commit d79537d2951dc6eabc7fabe6fa0760cb0f07c11d in qpid-broker-j's branch
refs/heads/7.0.x from [~k-wall]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=d79537d ]
QPID-7830: [Broker-J] [AMQP 0-8..0-91] Move caching responsubility to
virtualhost
(cherry picked from commit ddc519a551061c682877784068e755677e2c6313)
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Priority: Major
> Fix For: Future
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472215#comment-16472215
]
ASF subversion and git services commented on QPID-7830:
---
Commit 83c6dfebc3c815157ed3f04403a6968cc05eabae in qpid-broker-j's branch
refs/heads/7.0.x from [~k-wall]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=83c6dfe ]
QPID-7830: [Broker-J] [AMQP 0-8..0-91] Mechanically refactor AMQPShortString
introducing factory methods and hiding constructors
(cherry picked from commit 0a3dfc883b7af756fbdf076665eaae4ad8202bf7)
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Priority: Major
> Fix For: Future
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472217#comment-16472217
]
ASF subversion and git services commented on QPID-7830:
---
Commit 995d535bdea6fb5efcab9c165dbd9c807f94e72e in qpid-broker-j's branch
refs/heads/7.0.x from [~k-wall]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=995d535 ]
QPID-7830: [Broker-J] [AMQP 0-8..0-91] Cache AMQPShortStrings that relate to
exchanges/routing keys and header values (that are usually drawn from a small
domain) in a time/size bound cache.
Intent is to reduce the amount of tenured garbage produced when messages are
repeatedly sent to same destination. This should reduce frequency and length of
GC pauses.
(cherry picked from commit 7d7b50824ad9f2a99b7de034ef36a529129b00ac)
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Priority: Major
> Fix For: Future
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470550#comment-16470550
]
ASF subversion and git services commented on QPID-7830:
---
Commit 196b27065219b1283522bdb3a8f4fb1dda82ac16 in qpid-broker-j's branch
refs/heads/master from [~alex.rufous]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=196b270 ]
QPID-7830: [Broker-J] Fix failing test
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Priority: Major
> Fix For: Future
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466107#comment-16466107
]
ASF subversion and git services commented on QPID-7830:
---
Commit 5b6dfcd94ef64544040de53d38e2463d327ebcb1 in qpid-broker-j's branch
refs/heads/master from [~alex.rufous]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=5b6dfcd ]
QPID-7830: [Broker-J] Cache encoded and decoded values for 0-10 str8
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Priority: Major
> Fix For: Future
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463990#comment-16463990
]
Alex Rudyy commented on QPID-7830:
--
I reviewed the changes. They look good to me. Though, I would add additional
caching for 0-10 routing keys, exchange names (in DeliveryProperties),
contentType (MessageProperties) and 1-0 "to" field (in Properties) as it would
save us additional heap space and reduce frequency and length of GC pauses.
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Priority: Major
> Fix For: Future
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456290#comment-16456290
]
ASF subversion and git services commented on QPID-7830:
---
Commit ddc519a551061c682877784068e755677e2c6313 in qpid-broker-j's branch
refs/heads/master from [~k-wall]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=ddc519a ]
QPID-7830: [Broker-J] [AMQP 0-8..0-91] Move caching responsubility to
virtualhost
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Priority: Major
> Fix For: Future
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454415#comment-16454415
]
ASF subversion and git services commented on QPID-7830:
---
Commit 0a3dfc883b7af756fbdf076665eaae4ad8202bf7 in qpid-broker-j's branch
refs/heads/master from [~k-wall]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=0a3dfc8 ]
QPID-7830: [Broker-J] [AMQP 0-8..0-91] Mechanically refactor AMQPShortString
introducing factory methods and hiding constructors
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Priority: Major
> Fix For: Future
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454416#comment-16454416
]
ASF subversion and git services commented on QPID-7830:
---
Commit 7d7b50824ad9f2a99b7de034ef36a529129b00ac in qpid-broker-j's branch
refs/heads/master from [~k-wall]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=7d7b508 ]
QPID-7830: [Broker-J] [AMQP 0-8..0-91] Cache AMQPShortStrings that relate to
exchanges/routing keys and header values (that are usually drawn from a small
domain) in a time/size bound cache.
Intent is to reduce the amount of tenured garbage produced when messages are
repeatedly sent to same destination. This should reduce frequency and length of
GC pauses.
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Reporter: Keith Wall
>Priority: Major
> Fix For: Future
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220081#comment-16220081
]
Lorenz Quack commented on QPID-7830:
All this sounds awfully complicated.
What is wrong with something simpler like using a static class in broker-core
{{StringCache}} that uses a [guava
cache|https://github.com/google/guava/wiki/CachesExplained] with a [size
limit|https://github.com/google/guava/wiki/CachesExplained#size-based-eviction]
and [time based
eviction|https://github.com/google/guava/wiki/CachesExplained#timed-eviction]?
When a protocol layer then needs a String which we know might be repetitive
(like the cases mentioned above) it calls {{StringCache#getString()}} and guava
takes care of the rest.
This might be a naive approach but I would like to at least discuss why it is
not sufficient?
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
>Reporter: Keith Wall
> Fix For: Future
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218271#comment-16218271
]
Keith Wall commented on QPID-7830:
--
I was pondering the problem of which object(s) in the Broker would be
responsible for holding the caches of commonly used names such as destinations
or content-types. Global caches are awkward as they don't work well in the
embedded use-cases ({{String.intern}} falls into the category). I think that we
would want separate caches for different protocols - which will avoid polluting
AMQP 1.0 with AMQP 0-9..0-10 concerns.
I think we should have the VirtualHost additionally expose a set
ProtocolContext objects. ProtocolContext objects would be service loaded
using the Broker's existing pluggable pattern (factory etc). A protocol
implementations could one provide a factory if it wished. The protocol layer
itself would lookup its ProtocolContext from the VirtualHost. In a sense a
ProtocolObject would serve as a protocol specific 'singleton' scoped to the
virtualhost.
Turning back to the caching problem, in the 0-8..0-91 case,. its
ProtocolContext would expose a destination cache, content type cache etc. The
0-8 protocol layer, on the enqueue path, would consult these caches and
substitute the values from the cache if present. In the case of the
destination cache, we could make it respond to changes in the model so that
adding a queue or exchange automatically added a value to the destination cache
and removing one purged it. Perhaps for things like routingKeys, contentTypes
etc we keep a bounded LRU cache.
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
>Reporter: Keith Wall
> Fix For: Future
>
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055947#comment-16055947
]
Rob Godfrey commented on QPID-7830:
---
{quote}
On the AMQP 0-8..0-91, with things like contentType, path I wonder if keeping
the BasicContentHeaderProperties#_contentType as a separate AMQShortString is
useful.
{quote}
I agree - it would be better here to simply use Strings for all the string like
properties in the content header properties, and we can have a static list of
"common" types, and if the string is in that list we'll use the static value.
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
>Reporter: Keith Wall
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy with the same policy
> applying regardless of whether messages follow the on-line enqueue or
> recovery path. Note that in AMQP 1.0, values which are {{Symbols}} have
> their underlying String automatically interned.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (QPID-7830) Heap dominated by duplicates of common routing values / header values etc
[
https://issues.apache.org/jira/browse/QPID-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053913#comment-16053913
]
Rob Godfrey commented on QPID-7830:
---
One approach might be something like keeping a pool of string which are likely
to be reused (such as queue names, exchange names, etc), and checking in the
pool to find the shared copy. I'd be hesitant to try to pool routing keys
explicitly since by design these may pattern match... however for the common
routing-key=queue-name case, simply caching the queue names would be sufficient.
Historically we did cache short string values, but this was removed at some
point as I don't think it was adding much value.
> Heap dominated by duplicates of common routing values / header values etc
> -
>
> Key: QPID-7830
> URL: https://issues.apache.org/jira/browse/QPID-7830
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
>Reporter: Keith Wall
>
> When used for store and forwarding, in some use cases the Broker's heap can
> become dominated by duplicates of common values such as routing information
> (e.g. {{amq.direct}} or an application's queue name) or common header values
> (e.g a application/octet-stream or an application's user id).
> On the 0-8..0-91 paths, every enqueued message gets its own
> {{MessagePublishInfo}} referencing its own {{AMQShortString}} exchange and
> routing keys. For some use-cases, these are drawn from a small set. On the
> AMQP 1.0 path, {{Properties#to}} is an example. 0-10 is probably affected
> too.
> This unnecessarily increases the heap requirements of the Broker.
> The Broker should adopt a sensible intern/caching policy. Note that in AMQP
> 1.0, values which are {{Symbols}} have their underlying String automatically
> interned.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
