On 23 Feb, 16:05, "Simon Laws" <[EMAIL PROTECTED]> wrote:
> On 2/22/07, cdouglas <[EMAIL PROTECTED]> wrote:
>
>
>
>
>
> > The scenario is that I have 500 text files in a directory. Each file
> > contains up to 4000 XML documents of several different types just
> > appended one after another. The file size is between 20MB and 60MB.
> > Each individual XML file is about 10k bytes. I get the out of memory
> > error roughly after 5-6 files. If I restart the script, it will
> > process 5-6 more and then throw another memory error.
>
> > My script goes through the directory and loads each file one by one
> > into a string. I explode it so that I get each XML doc into its own
> > array element. I then create a new SDO object using the correct XSD
> > for the XML type and load the xml string from the array. I pull out
> > data, massage it, and then write it to a MySQL database. I then clean
> > up the created object, strings and go onto the next array element.
>
> > Using the PHP memory_get_usage, it reports only a little more than the
> > size of the current file I am processing, which is up to 60MB. My PHP
> > mem limit is set to 512 MB and the machine has 8 GB of RAM.
>
> > Thanks
> > Chris
>
> Thanks for that Chris
>
> I started testing with very small files which highlights performance issues
> we have with loading the files in the first place and memory leak
> performance which I passed over to the tuscany team. I'll up my file sizes
> and make them like you describe and see what happens now..
>
> Regards
>
> Simon
Hi Chris,
I've done a little playing about with this and managed to reproduce
the kinds of results you are getting. The script I have loads an XSD
into the XML DAS and then repeatedly loads a 15K XML file. Observing
this running with the SDO code as it stands you can see the memory
consumed by the PHP process (when looking at the windows task manager)
shooting up as the run progresses. The final result on my machine is
that it fails with an error around loop number 9500. Fortunately
various people are looking at memory leaks at the moment and Caroline
pointed my in the direction of a known problem in SDO_DataObject.cpp
static void sdo_do_object_free_storage(void *object TSRMLS_DC)
{
sdo_do_object *my_object;
char *space;
char *class_name;
my_object = (sdo_do_object *)object;
zend_hash_destroy(my_object->zo.properties);
FREE_HASHTABLE(my_object->zo.properties);
if (my_object->zo.guards) {
zend_hash_destroy(my_object->zo.guards);
FREE_HASHTABLE(my_object->zo.guards);
}
/* just release the reference, and the reference counting will kick
in */
if (my_object->dop) {
try {
my_object->dop->setUserData((void
*)SDO_USER_DATA_EMPTY);
/* causes an Access Violation :-( */
//my_object->dop = NULL;
} catch (SDORuntimeException e) {
class_name = get_active_class_name(&space TSRMLS_CC);
php_error(E_WARNING, "%s%s%s(): internal error - caught
exception
freeing DataObject",
class_name, space,
get_active_function_name(TSRMLS_C));
}
}
efree(object);
}
You will note that the line that releases the data object memory is
commented out due to some access violation. I took my life in my hands
and reinstated this and my test ran much more happily, i.e. it didn't
kill my machine by consuming all the RAM, and it ran to completion.
Caroline is tracking down why this has been flagged as problematic and
will provide a fix in due course. In the mean time it would be
interesting to see if it has the same positive effect for you as it
did for me. There is of course the chance that it will just fail with
the mysterious access violation. Anyhow if you are not in a position
to play with this watch this space and I'll let you know when the
proper fix is in place.
Regards
Simon
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"phpsoa" group.
To post to this group, send email to phpsoa@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.co.uk/group/phpsoa?hl=en
-~----------~----~----~----~------~----~------~--~---
