Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Markos: can you approve this now? -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Review: Approve -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
The proposal to merge lp:~zorba-coders/zorba/feature-mem_size into lp:zorba has been updated. Status: Needs review = Approved For more details, see: https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Validation queue starting for merge proposal. Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-mem_size-2013-02-08T20-33-54.456Z/log.html -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
The attempt to merge lp:~zorba-coders/zorba/feature-mem_size into lp:zorba failed. Below is the output from the failed tests. CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:275 (message): Validation queue job feature-mem_size-2013-02-08T20-33-54.456Z is finished. The final status was: 1 tests did not succeed - changes not commited. Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
The proposal to merge lp:~zorba-coders/zorba/feature-mem_size into lp:zorba has been updated. Status: Approved = Needs review For more details, see: https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
The proposal to merge lp:~zorba-coders/zorba/feature-mem_size into lp:zorba has been updated. Status: Needs review = Approved For more details, see: https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Validation queue starting for merge proposal. Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-mem_size-2013-02-08T23-51-38.613Z/log.html -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
FYI: I had to remove the XQuery test for an item's size because the expected results can't be hard-coded since they're platform-dependent. For example, I get totally different values on Mac OS X. The reason the C++ unit tests for mem_sizeof() work (are platform-independent) is because the expected results are computed rather than hard-coded. -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Validation queue job feature-mem_size-2013-02-08T23-51-38.613Z is finished. The final status was: All tests succeeded! -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
The proposal to merge lp:~zorba-coders/zorba/feature-mem_size into lp:zorba has been updated. Status: Approved = Merged For more details, see: https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
I think everything works now. -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
On Jan 10, 2013, at 2:28 PM, Markos Zaharioudakis markos...@yahoo.com wrote: The problem is infinite recursion. If I ask for the size of a tree, it's the sum of all the nodes. If the root node includes the size of the tree, then it will recurse infinitely. I don't understand the infinite recursion problem. What I have in mind is add an alloc_size() method to the XmlNode class, which will return 0 if the node is not a root, otherwise returm mem_sizeof(XmlTree). If it returns 0, then you could never get the size of an XmlNode (or any class derived from XmlNode) independently of the tree. IMHO, you should be able to get the mem_sizeof any node (which would include its child nodes, if any) independently of the tree. If you explicitly ask for the mem_sizeof an XmlTree explicitly, then you'd get the size of the tree object itself plus all the nodes. - Paul -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
On Jan 10, 2013, at 1:56 PM, Markos Zaharioudakis markos...@yahoo.com wrote:
Here is the 1st problem: inside mem_sizeof(const XmlNode t), sizeof(T) (or
sizeof(t)) should return 52 if t is actually an ElementNode. Of course, this
is not possible since sizeof(T) is computed statically (i.e., at compile
time).
Oh. I get it now. That's a serious problem. One solution I can think of is
to have size_traitsT have another member function... something like
static_sizeof(T const). Unfortunately the only way that can work is if every
class in a class hierarchy implements a virtual member function static_size()
whose implementation is ALWAYS:
size_t static_size() const {
return sizeof( *this );
}
As for why the template trick doesn't work, I don't know yet. Perhaps it's
because XmlNode doesn't implement its own alloc_size() but instead inherits it
from Item (though I could have sworn I tested this and it worked).
- Paul
--
https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764
Your team Zorba Coders is subscribed to branch lp:zorba.
--
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
On Dec 17, 2012, at 2:44 AM, Markos Zaharioudakis markos...@yahoo.com wrote:
size:size(abfoo/b/a) returns 101. This is wrong because just
sizeof(ElementNode) is 52 and the tree abfoo/b/a consists of 2
element nodes, one text node and one (hidden) attribute node.
The reason for this wrong result is in this struct:
templatetypename T
struct size_traitsT*,false {
static size_t alloc_sizeof( T *p ) {
return p ? mem_sizeof( *p ) : 0;
}
};
Basically, an InternalNode has a vector of XmlNode*. So, the above struct is
instantiated with T being XmlNode. Then, mem_sizeof(XmlNode) is called
returning just 20 (== sizeof(XmlNode)).
I've looked at this a bit more and I still don't understand why it's wrong.
Ignoring everything else, the alloc_size() specialization for pointers seems
correct.
For a vectorXmlNode*, it should iterate through the vector's elements. For
each element, it should call the above alloc_sizeof() for XmlNode*. That in
turn should call mem_sizeof(XmlNode) which is sizeof(XmlNode) +
alloc_sizeof(XmlNode). alloc_sizeof(XmlNode) turns into
size_traitsXmlNode::alloc_sizeof(t).
XmlNode is derived from store::Item that has a virtual alloc_size(). The
specialization of size_traits for classes that have an alloc_size() member
function is called. Therefore, I don't see where the mistake is.
- Paul
--
https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764
Your team Zorba Coders is subscribed to branch lp:zorba.
--
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
On Dec 17, 2012, at 2:44 AM, Markos Zaharioudakis markos...@yahoo.com wrote:
size:size(abfoo/b/a) returns 101. This is wrong because just
sizeof(ElementNode) is 52 and the tree abfoo/b/a consists of 2 element
nodes, one text node and one (hidden) attribute node.
The reason for this wrong result is in this struct:
templatetypename T
struct size_traitsT*,false {
static size_t alloc_sizeof( T *p ) {
return p ? mem_sizeof( *p ) : 0;
}
};
Basically, an InternalNode has a vector of XmlNode*. So, the above struct is
instantiated with T being XmlNode. Then, mem_sizeof(XmlNode) is called
returning just 20 (== sizeof(XmlNode)).
I've looked at this a bit more and I still don't understand why it's wrong.
Ignoring everything else, the alloc_size() specialization for pointers seems
correct.
For a vectorXmlNode*, it should iterate through the vector's elements. For
each element, it should call the above alloc_sizeof() for XmlNode*. That in
turn should call mem_sizeof(XmlNode) which is sizeof(XmlNode) +
alloc_sizeof(XmlNode).
Here is the 1st problem: inside mem_sizeof(const XmlNode t), sizeof(T) (or
sizeof(t)) should return 52 if t is actually an ElementNode. Of course, this is
not possible since sizeof(T) is computed statically (i.e., at compile time).
alloc_sizeof(XmlNode) turns into
size_traitsXmlNode::alloc_sizeof(t).
And here is the 2nd problem: alloc_sizeof(XmlNode) turns into
size_traitsXmlNode, false::alloc_sizeof(t), which returns 0. Somehow, the
trick with the has_alloc_size class does not work.
XmlNode is derived from store::Item that has a virtual alloc_size(). The
specialization of size_traits for classes that have an alloc_size() member
function is called. Therefore, I don't see where the mistake is.
- Paul
Try stepping through this example with gdb, and you will see it yourself.
--
https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764
Your team Zorba Coders is subscribed to branch lp:zorba.
--
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
The size of XmlTree is not counted. Right: because it's shared by all nodes in the tree. An instance of XmlTree is shared by all nodex in the same xml tree. Maybe the size of the XmlTree should be attribute only to the root node of the tree. The problem is infinite recursion. If I ask for the size of a tree, it's the sum of all the nodes. If the root node includes the size of the tree, then it will recurse infinitely. - Paul I don't understand the infinite recursion problem. What I have in mind is add an alloc_size() method to the XmlNode class, which will return 0 if the node is not a root, otherwise returm mem_sizeof(XmlTree). -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
On Dec 16, 2012, at 8:43 PM, Markos Zaharioudakis markos...@yahoo.com wrote: The memory size of some std containers is underestimated, because the book- keeping overhead is not taken into account. I understand we cannot be 100% exact here, but should we try to consider some common std implementations and measure the mem size accordingly? For example, std::map is typically implemented as a red-black tree, which should add at least 2 pointers (or offsets) per entry into its memory size. OK, I added sizeof(void*) * 2. I guess the unordered_map (and set) should have 1 extra pointer per entry and 1 extra pointer per hash bucket? -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
The function alloc_sizeof( rstringRepType const s ) does not count the book-keeping overhead for the various kinds of strings. For example, the default rep (which is used for zstrings) adds 12 bytes oberhead (on 32-bit machines). I changed it to: return s.capacity() + (s.is_shared() ? 0 : sizeof( RepType )); This is fine with me, but there are many other things that are shared and yet their sharing is not taken into account (e.g. qname items in the qname pool, or atomic items shared among the nodes of copied xml trees). I think we should have a consistent solution about what to do with shared things. There are several options, but I will let Matthias decide about this. Also, I think the null-terminating byte is not counted. The alloc_size is its capacity. The null byte is irrelevant. capacity() does not include the null-terminating byte. The actual number of bytes allocated is capacity()+1 (plus sizeof(RepType)). -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
On Dec 16, 2012, at 8:43 PM, Markos Zaharioudakis markos...@yahoo.com wrote: The memory size of some std containers is underestimated, because the book-keeping overhead is not taken into account. I understand we cannot be 100% exact here, but should we try to consider some common std implementations and measure the mem size accordingly? For example, std::map is typically implemented as a red-black tree, which should add at least 2 pointers (or offsets) per entry into its memory size. OK, I added sizeof(void*) * 2. On Dec 16, 2012, at 8:53 PM, Markos Zaharioudakis markos...@yahoo.com wrote: I think we should define Base16::alloc_size() and Base64::alloc_size() methods. The Base16::alloc_size() method should then be used in HexBinaryItem::alloc_size() (which is not quite correct right now). StructuralAnyUriItem::alloc_size() does not count theEncodedValue StreamableStringItem::alloc_size() does not count theStreamableDependent Fixed. On Dec 16, 2012, at 10:00 PM, Markos Zaharioudakis markos...@yahoo.com wrote: The function alloc_sizeof( rstringRepType const s ) does not count the book-keeping overhead for the various kinds of strings. For example, the default rep (which is used for zstrings) adds 12 bytes oberhead (on 32-bit machines). I changed it to: return s.capacity() + (s.is_shared() ? 0 : sizeof( RepType )); Also, I think the null-terminating byte is not counted. The alloc_size is its capacity. The null byte is irrelevant. On Dec 17, 2012, at 2:52 AM, Markos Zaharioudakis markos...@yahoo.com wrote: The size of XmlTree is not counted. Right: because it's shared by all nodes in the tree. An instance of XmlTree is shared by all nodex in the same xml tree. Maybe the size of the XmlTree should be attribute only to the root node of the tree. The problem is infinite recursion. If I ask for the size of a tree, it's the sum of all the nodes. If the root node includes the size of the tree, then it will recurse infinitely. - Paul -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
size:size(abfoo/b/a) returns 101. This is wrong because just
sizeof(ElementNode) is 52 and the tree abfoo/b/a consists of 2 element
nodes, one text node and one (hidden) attribute node.
The reason for this wrong result is in this struct:
templatetypename T
struct size_traitsT*,false {
static size_t alloc_sizeof( T *p ) {
return p ? mem_sizeof( *p ) : 0;
}
};
Basically, an InternalNode has a vector of XmlNode*. So, the above struct is
instantiated with T being XmlNode. Then, mem_sizeof(XmlNode) is called
returning just 20 (== sizeof(XmlNode)).
--
https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764
Your team Zorba Coders is subscribed to branch lp:zorba.
--
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
The size of XmlTree is not counted. An instance of XmlTree is shared by all nodex in the same xml tree. Maybe the size of the XmlTree should be attribute only to the root node of the tree. -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
The memory size of some std containers is underestimated, because the book-keeping overhead is not taken into account. I understand we cannot be 100% exact here, but should we try to consider some common std implementations and measure the mem size accordingly? For example, std::map is typically implemented as a red-black tree, which should add at least 2 pointers (or offsets) per entry into its memory size. -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
I think we should define Base16::alloc_size() and Base64::alloc_size() methods. The Base16::alloc_size() method should then be used in HexBinaryItem::alloc_size() (which is not quite correct right now). StructuralAnyUriItem::alloc_size() does not count theEncodedValue StreamableStringItem::alloc_size() does not count theStreamableDependent -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
My preference would be to make the default alloc_size() function raise an error, instead of returning 0. Then, every class whose mem size may need to be counted should have its own alloc_size() method, even if it does not have any heap-allocated memory. I know this would be more intrusive, but I think it would be more robust as well. However, this is just my preference; I will let you and Matthias decide on this. -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
The function alloc_sizeof( rstringRepType const s ) does not count the book-keeping overhead for the various kinds of strings. For example, the default rep (which is used for zstrings) adds 12 bytes oberhead (on 32-bit machines). Also, I think the null-terminating byte is not counted. -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Review: Needs Fixing -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
I added code to count the size of namespace bindings for element nodes. -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
I did some testing and the numbers are much better/consistent. But I didn't change anything. It's at test/rbkt/Queries/zorba/item/size.xq. This is impossible to test for since things like sizeof(N) (where N is some native type like int) and sizeof(S) (where S is some struct/class type) varies by platform and compiler due to struct layout and padding. -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Review: Needs Fixing I did some testing and the numbers are much better/consistent. I have started a test to make sure we keep this on the radar and understand if something regresses. It's at test/rbkt/Queries/zorba/item/size.xq. Paul, could you please extend this test with at least one example for all atomic types and some examples for nodes, json objects, and json arrays? Once we have this test, the branch should be merged and we improve the size computation as we go. -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Why does ztd::alloc_sizeof( theKeys ) in src/store/naive/json_items.cpp:245 return 0? Because a specialization for HashMap wasn't ever defined. I really wish we'd get away from using our own non-standard hashmaps. There really is no point. -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Review: Needs Information Why does ztd::alloc_sizeof( theKeys ) in src/store/naive/json_items.cpp:245 return 0? This doesn't seem to be true. -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
I added Item::mem_size(). -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Review: Needs Information Can we find a way to expose the functionality in the public C++ api? -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Can we find a way to expose the functionality in the public C++ api? There are two choices: 1. Expose the entire mem_sizeof.h header and add: size_t alloc_size() const; on the public Item. 2. Add: size_t mem_size() const; on the public Item that simply calls ztd::mem_sizeof() on the store::Item. -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Can we find a way to expose the functionality in the public C++ api? There are two choices: 1. Expose the entire mem_sizeof.h header and add: size_t alloc_size() const; on the public Item. 2. Add: size_t mem_size() const; on the public Item that simply calls ztd::mem_sizeof() on the store::Item. I prefer option 2. Could you please do that and mention it in the ChangeLog? -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Review: Needs Fixing I have added a new XQuery module (http://www.zorba-xquery.com/modules/item) which currently provides one function x:allocated-size. The function invokes Item::alloc_size. Review comments - The function Item::alloc_size should be exposed in the external C++ api. - The function alloc_size should also be implemented for JSONiq items. - In file included from /home/mbrantner/zorba/sandbox/src/unit_tests/test_mem_sizeof.cpp:21:0: /home/mbrantner/zorba/sandbox/src/util/mem_sizeof.h:177:58: warning: all member functions in class ‘zorba::ztd::has_alloc_sizeT’ are private - returns zero for integer: import module namespace x = http://www.zorba-xquery.com/modules/item;; x:allocated-size(1) -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Please note the distinction between alloc_sizeof() and mem_sizeof(). The former obtains the *additionally* dynamically allocated memory for an object of type T. An 'int' has no additional memory beyond the size of the 'int' itself, so the result of 0 is correct. Why do you want to expose *only* alloc_size()? It's kind of awkward without mem_sizeof(). BTW: alloc_size() isn't really meant to be called explicitly. You *can* do it and it will give you the correct answer, but it's not the answer you really want. Again, note the distinction between alloc_sizeof() and mem_sizeof(). -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Paul J. Lucas has proposed merging lp:~zorba-coders/zorba/feature-mem_size into
lp:zorba.
Requested reviews:
Paul J. Lucas (paul-lucas)
For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764
Added framework for calculating the total memory used by a data structure and
using it with store::Item.
--
https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764
Your team Zorba Coders is subscribed to branch lp:zorba.
=== modified file 'src/store/api/item.h'
--- src/store/api/item.h 2012-07-12 17:29:55 +
+++ src/store/api/item.h 2012-07-13 00:33:23 +
@@ -104,6 +104,7 @@
void removeReference();
+ virtual size_t alloc_size() const;
/* --- General Methods for Items - */
=== modified file 'src/store/naive/atomic_items.cpp'
--- src/store/naive/atomic_items.cpp 2012-07-12 17:29:55 +
+++ src/store/naive/atomic_items.cpp 2012-07-13 00:33:23 +
@@ -43,6 +43,7 @@
#include tree_id.h
#include util/ascii_util.h
+#include util/mem_sizeof.h
#include util/string_util.h
#include util/utf8_util.h
@@ -381,6 +382,14 @@
}
+size_t UserTypedAtomicItem::alloc_size() const
+{
+ return AtomicItem::alloc_size()
++ ztd::alloc_sizeof( theBaseItem )
++ ztd::alloc_sizeof( theTypeName );
+}
+
+
/***
class UntypedAtomicItem
/
@@ -571,6 +580,12 @@
}
+size_t UntypedAtomicItem::alloc_size() const
+{
+ return AtomicItem::alloc_size() + ztd::alloc_sizeof( theValue );
+}
+
+
bool UntypedAtomicItem::equals(
const store::Item* other,
long timezone,
@@ -679,6 +694,15 @@
}
+size_t QNameItem::alloc_size() const
+{
+ return AtomicItem::alloc_size()
++ ztd::alloc_sizeof( theNamespace )
++ ztd::alloc_sizeof( thePrefix )
++ ztd::alloc_sizeof( theLocal );
+}
+
+
store::Item* QNameItem::getType() const
{
return GET_STORE().theSchemaTypeNames[store::XS_QNAME];
@@ -816,6 +840,12 @@
}
+size_t NotationItem::alloc_size() const
+{
+ return ztd::mem_sizeof( *theQName );
+}
+
+
store::Item* NotationItem::getType() const
{
return GET_STORE().theSchemaTypeNames[store::XS_NOTATION];
@@ -869,6 +899,12 @@
}
+size_t AnyUriItem::alloc_size() const
+{
+ return AtomicItem::alloc_size() + ztd::alloc_sizeof( theValue );
+}
+
+
bool AnyUriItem::getEBV() const
{
return ! (theValue == );
@@ -1609,10 +1645,21 @@
}
}
+size_t StructuralAnyUriItem::alloc_size() const
+{
+ return AnyUriItem::alloc_size() + ztd::alloc_sizeof( theOrdPath );
+}
+
/***
class StringItem
/
+
+size_t StringItem::alloc_size() const
+{
+ return AtomicItem::alloc_size() + ztd::alloc_sizeof( theValue );
+}
+
store::Item* StringItem::getType() const
{
return GET_STORE().theSchemaTypeNames[store::XS_STRING];
@@ -2402,6 +2449,12 @@
class DecimalItem
/
+size_t DecimalItem::alloc_size() const
+{
+ return AtomicItem::alloc_size() + ztd::alloc_sizeof( theValue );
+}
+
+
store::Item* DecimalItem::getType() const
{
return GET_STORE().theSchemaTypeNames[store::XS_DECIMAL];
@@ -2451,6 +2504,14 @@
class IntegerItemImpl
/
+#ifdef ZORBA_WITH_BIG_INTEGER
+size_t IntegerItemImpl::alloc_size() const
+{
+ return ztd::alloc_size( theValue );
+}
+#endif /* ZORBA_WITH_BIG_INTEGER */
+
+
long IntegerItemImpl::compare( Item const *other, long,
const XQPCollator* ) const {
try
@@ -2557,6 +2618,15 @@
/***
class NonPositiveIntegerItem
/
+
+#ifdef ZORBA_WITH_BIG_INTEGER
+size_t NonPositiveIntegerItem::alloc_size() const
+{
+ return ztd::alloc_size( theValue );
+}
+#endif /* ZORBA_WITH_BIG_INTEGER */
+
+
long NonPositiveIntegerItem::compare( Item const *other, long,
const XQPCollator* ) const {
try
@@ -2669,6 +2739,14 @@
/***
class NonNegativeIntegerItem
/
+
+#ifdef ZORBA_WITH_BIG_INTEGER
+size_t NonNegativeIntegerItem::alloc_size() const
+{
+ return ztd::alloc_size( theValue );
+}
+#endif /* ZORBA_WITH_BIG_INTEGER */
+
long NonNegativeIntegerItem::compare( Item const *other, long,
const XQPCollator* ) const {
try
@@ -3330,6 +3408,12 @@
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Review: Approve -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
The proposal to merge lp:~zorba-coders/zorba/feature-mem_size into lp:zorba has been updated. Status: Needs review = Approved For more details, see: https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Validation queue starting for merge proposal. Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-mem_size-2012-07-13T01-17-04.607Z/log.html -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Validation queue job feature-mem_size-2012-07-13T01-17-04.607Z is finished. The final status was: All tests succeeded! -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
Voting does not meet specified criteria. Required: Approve 1, Disapprove 1, Needs Fixing 1, Pending 1. Got: 1 Approve. -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
[Zorba-coders] [Merge] lp:~zorba-coders/zorba/feature-mem_size into lp:zorba
The proposal to merge lp:~zorba-coders/zorba/feature-mem_size into lp:zorba has been updated. Status: Approved = Needs review For more details, see: https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 -- https://code.launchpad.net/~zorba-coders/zorba/feature-mem_size/+merge/114764 Your team Zorba Coders is subscribed to branch lp:zorba. -- Mailing list: https://launchpad.net/~zorba-coders Post to : zorba-coders@lists.launchpad.net Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp
