Now that we've gotten past (at least for a little while) the 10,000 user replication issue, another serious XML-RPC issue as surfaced, but this time appears to be with reading a large database.
In these particular cases: - XX-6637 "Registrations page was showing no registrations, when more than 2500 users were registered" - XX-6639 "registration.xml with 10,000 users registered, did not replicate on the distributed machine" an XML-RPC request is made to sipXsupervisor to read the Registration IMDB. I have traced through and found that the issue is not with reading all of the records from the IMDB but in actually building the XML-RPC response that will contain all of the records. The XML-RPC response is essentially a UtlString built by traversing through the result set from the IMDB read and "appending" each record + appropriate tags around each field. The result set given by the IMDB read(which is passed to XmlRpcResponse) is a UtlSList containing a Hash Map for each record. The UtlString append method (as I understand it), will, if the current memory chunk is not big enough: - Allocate a new larger chunk of memory. But only large enough to hold the new appended string. - Copy the existing string to this new chunk of memory. - Copy the characters to append to this new chunk of memory. - Delete the old chunk of memory and return with the new. In our case, this is extremely inneficient because there is a large amount of calls to the UtlString append that require this re-allocation of memory. In my testing, I was trying to read (through XML-RPC) the permissions database on my system. This database contains almost 91,000 records for a system with 10,000 users. I set the XML-RPC timeout to 3 minutes. The XML-RPC timeout occurred before the response was built. As a matter of fact, sipXsupervisor after more than 5 minutes never finished building the result. For the 4.0 stream, I'm planning on modifying the building of this UtlString (in XmlRpcBody.cpp) by determing the number of entries (i.e. records) in the UtlSList and setting the capacity to (# of records * 650 bytes). I've looked at the record size of a Registration and figured with the addition of the XML tags that 650 bytes per record should be good enough. I've tested this change with the same permissions database (~91,000 records) as before and was able to return an XML-RPC response in about 6 seconds (certainly better than > 5 minutes). In the main stream, however, I'd like some thoughts on a better/more generic way of speculating on the size of the response that's going to be built so that we can reduce/eliminate the UtlString resizing. Please note that this would still not be the ideal solution. The underlying problem of using XML-RPC in these scenarios will still exist and a "new" approach will eventually be required. Thoughts? Raymond _______________________________________________ sipx-dev mailing list [email protected] List Archive: http://list.sipfoundry.org/archive/sipx-dev Unsubscribe: http://list.sipfoundry.org/mailman/listinfo/sipx-dev sipXecs IP PBX -- http://www.sipfoundry.org/
