Yes, trying with the latest stable version (3.4.2) would be useful, if you don't mind. There were some lowlevel garbage collection problems that were fixed in the 3.4.2 release; these problems manifested themselves in a variety of ways.
I'm not saying this is the issue, but it wouldn't hurt to verify that your problem can be reproduced in 3.4.2. thanks, -- Scott On 05/17/2011 12:22 PM, Scott Hammel wrote: > I'm pretty sure it is 3.4.0 (from files on the server it looks like an > August 2010 build. The server is in a totally isolated network with > nothing with GUI support that can hit the admin tools). > > Tomcat is the version bundled with the Fedora installer. > > Would you like me to be sure I'm running at the latest version and try > the test scripts again before you go forward? > > Scott > > On 05/17/2011 12:45 PM, Scott Prater wrote: >> Thanks, Scott. I'll try to reproduce the problem in my environment, >> Fedora 3.4.2. >> >> Can you tell me what version of Fedora and Tomcat (or other webapp >> server) you're using? >> >> -- Scott >> >> On 05/17/2011 11:08 AM, Scott Hammel wrote: >>> Hey, Scott, >>> >>> Thanks for responding. I'm more a C/C++ programmer and not a Java >>> programmer (though I sometimes play one on the Internet), so I'm just >>> guessing on the array bounds -- feels like something incrementing an int >>> into the sign bit, though I'd think Java would throw some array bounds >>> exception before that happened. Figured I'd do a little math later maybe >>> to test my hypothesis. >>> >>> Recall, this was all in a 32-bit environment. I really hope it is a >>> non-issue and something I'm doing in the end. Note disseminating the >>> datastream content directly appears to work OK, which confuses me a >>> little, though I haven't looked to see if the code for that does things >>> differently. >>> >>> Anyway, here's a series of commands (extracted from my test scripts) >>> that should reproduce the problem: >>> >>> mkdir /usr/fedora/tomcat/webapps/ROOT/ingestpool >>> mkdir /tmp/fedrun >>> dir=/tmp/fedrun >>> pid=test:pid01 >>> >>> dd if=/dev/urandom >>> of=/usr/fedora/tomcat/webapps/ROOT/ingestpool/sample.bin bs=1M count=400 >>> >>> ./makefoxml $pid http://localhost:8080/ingestpool/sample.bin> >>> $dir/sample.xml >>> >>> /usr/fedora/client/bin/fedora-ingest.sh f $dir/sample.xml >>> info:fedora/fedora-system:FOXML-1.1 localhost:8080 fedoraAdmin<insert >>> pwd here> http >>> >>> wget -O $dir/export.xml --auth-no-challenge --http-user=fedoraAdmin >>> --http-password=<insert pwd here> >>> http://localhost:8080/fedora/objects/$pid/export?context=archive >>> >>> Note: I use the REST call via a wget rather than the provided export >>> client scripts because it looks to me from the Java heap explosion that >>> the export scripts must end up doing the export via the SOAP API. >>> -- >>> The content of makefoxml: >>> >>> #!/bin/bash >>> >>> #usage: makefoxml<pid> <refurl> >>> #escape slashes off the URL >>> RF=${2//\//\\/} >>> #if you need to escape ampersands as well, uncomment this: >>> #RF=${RF//'&'/'\&'} >>> >>> # make substitutions .... >>> sed ' >>> s/PID=""/PID="'"$1"'"/ >>> s/rdf:about=""/rdf:about="info:fedora\/'"$1"'"/ >>> s/dc:identifier>/dc:identifier>'"$1"'/ >>> s/REF=""/REF="'"${RF}"'"/ >>> '< "foxml_tpl.xml" >>> >>> -- >>> The content of foxml_tmp.xml (the sed script above does the edits noted >>> in the xml comments in this template): >>> >>> <?xml version="1.0" encoding="UTF-8"?> >>> <!-- following element: set the PID attribute --> >>> <foxml:digitalObject VERSION="1.1" PID="" >>> xmlns:foxml="info:fedora/fedora-system:def/foxml#" >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>> xsi:schemaLocation="info:fedora/fedora-system:def/foxml# >>> http://www.fedora.info/definitions/1/0/foxml1-1.xsd";> >>> >>> <foxml:objectProperties> >>> <foxml:property NAME="info:fedora/fedora-system:def/model#state" VALUE="A"/> >>> <foxml:property NAME="info:fedora/fedora-system:def/model#label" VALUE=""/> >>> <foxml:property NAME="info:fedora/fedora-system:def/model#ownerId" >>> VALUE="fedoraAdmin"/> >>> </foxml:objectProperties> >>> >>> <foxml:datastream CONTROL_GROUP="X" ID="RELS-EXT"> >>> <foxml:datastreamVersion >>> FORMAT_URI="info:fedora/fedora-system:FedoraRELSExt-1.0" >>> ID="RELS-EXT.0" LABEL="RDF Statements about this Object" >>> MIMETYPE="application/rdf+xml"> >>> <foxml:xmlContent> >>> <rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/"; >>> >>> xmlns:fedora="info:fedora/fedora-system:def/relations-external#" >>> xmlns:fedora-model="info:fedora/fedora-system:def/model#" >>> xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/"; >>> xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"; >>> xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#";> >>> <!-- following element: put the PID as the value for the rdf:about >>> attribute --> >>> <rdf:description rdf:about=""> >>> </rdf:description> >>> </rdf:RDF> >>> </foxml:xmlContent> >>> </foxml:datastreamVersion> >>> </foxml:datastream> >>> >>> <foxml:datastream CONTROL_GROUP="X" ID="DC" STATE="A" VERSIONABLE="true"> >>> <foxml:datastreamVersion ID="DC.0" LABEL="Dublin Core Record" >>> MIMETYPE="text/xml"> >>> <foxml:xmlContent> >>> <oai_dc:dc xmlns:dc="http://purl.org/dc/elements/1.1/"; >>> xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/"; >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>> xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ >>> http://www.openarchives.org/OAI/2.0/oai_dc.xsd";> >>> <dc:title></dc:title> >>> <dc:creator>Test Program</dc:creator> >>> <dc:description>A test object</dc:description> >>> <!-- following element: put the PID between the tags --> >>> <dc:identifier></dc:identifier> >>> </oai_dc:dc> >>> </foxml:xmlContent> >>> </foxml:datastreamVersion> >>> </foxml:datastream> >>> >>> <foxml:datastream CONTROL_GROUP="M" ID="Content" STATE="A"> >>> <foxml:datastreamVersion ID="Content.0" LABEL="This is the object >>> content" MIMETYPE=" application/octet-stream"> >>> <!-- following element: put the URL to the content file as the value for >>> the REF attribute --> >>> <!-- must be an http URL, e.g., >>> http://localhost:8080/ingestpool/foxmldoc.xml --> >>> <!-- I just create a directory "ingestpool" under >>> /usr/fedora/tomcat/webapps/ROOT and put the files there --> >>> <foxml:contentLocation REF="" TYPE="URL" /> >>> </foxml:datastreamVersion> >>> </foxml:datastream> >>> >>> >>> </foxml:digitalObject> >>> >>> >>> >>> On 05/17/2011 10:00 AM, Scott Prater wrote: >>>> Scott, >>>> >>>> Can you come up with a test case that confirms this limitation? If you >>>> can provide one, I'll open up a JIRA ticket for the issue. >>>> >>>> thanks, >>>> >>>> -- Scott >>>> >>>> On 05/16/2011 10:45 AM, Scott Hammel wrote: >>>>> Oh, I think I see: last line of the serializer's serialize function does >>>>> this: >>>>> bytes.toByteArray() >>>>> where bytes is a ByteArrayOutputStream >>>>> >>>>> I *think* the max size of an array index in Java (32-bit) is >>>>> 2,147,483,647 (i.e., 2^31 - 1, max value of a java int). So, this >>>>> function will throw an exception if a datastream "archive" export is> >>>>> ~2 GB. >>>>> >>>>> scott >>>>> >>>>> On 05/16/2011 11:00 AM, Scott Hammel wrote: >>>>>> Hi, >>>>>> >>>>>> Running some export tests using Fedora's REST export API, I get a >>>>>> negative array index Java exception when doing an "archive" export of an >>>>>> object at around 400 MB (>320 MB,< 450 MB). >>>>>> >>>>>> Fedora is version 3.4 something; running on 32-bit CentOS 5.5, Sun Java >>>>>> 1.6, 21 >>>>>> >>>>>> Is it just me or has anyone else seen something like that? >>>>>> >>>>>> Thanks, >>>>>> Scott >>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>>> Achieve unprecedented app performance and reliability >>>>>> What every C/C++ and Fortran developer should know. >>>>>> Learn how Intel has extended the reach of its next-generation tools >>>>>> to help boost performance applications - inlcuding clusters. >>>>>> http://p.sf.net/sfu/intel-dev2devmay >>>>>> _______________________________________________ >>>>>> Fedora-commons-users mailing list >>>>>> Fedora-commons-users@lists.sourceforge.net >>>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Achieve unprecedented app performance and reliability >>>>> What every C/C++ and Fortran developer should know. >>>>> Learn how Intel has extended the reach of its next-generation tools >>>>> to help boost performance applications - inlcuding clusters. >>>>> http://p.sf.net/sfu/intel-dev2devmay >>>>> _______________________________________________ >>>>> Fedora-commons-users mailing list >>>>> Fedora-commons-users@lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>> >>> ------------------------------------------------------------------------------ >>> Achieve unprecedented app performance and reliability >>> What every C/C++ and Fortran developer should know. >>> Learn how Intel has extended the reach of its next-generation tools >>> to help boost performance applications - inlcuding clusters. >>> http://p.sf.net/sfu/intel-dev2devmay >>> _______________________________________________ >>> Fedora-commons-users mailing list >>> Fedora-commons-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >> > > > ------------------------------------------------------------------------------ > Achieve unprecedented app performance and reliability > What every C/C++ and Fortran developer should know. > Learn how Intel has extended the reach of its next-generation tools > to help boost performance applications - inlcuding clusters. > http://p.sf.net/sfu/intel-dev2devmay > _______________________________________________ > Fedora-commons-users mailing list > Fedora-commons-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users -- Scott Prater Library, Instructional, and Research Applications (LIRA) Division of Information Technology (DoIT) University of Wisconsin - Madison pra...@wisc.edu ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ Fedora-commons-users mailing list Fedora-commons-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
