Hi Janna, The "too many open files" error is troubling. I'd like to try to reproduce it. I noticed in your log snippet there are two back-to-back calls to getNextPID(). Can you describe the sequence of calls you're making to the REST API in more detail? Are these *all* calls to that method, or are there calls in between which attempt to ingest the objects? If so, about how many getNextPID calls are you making between each ingest request? Are there any other REST API calls in between?
Thanks, Chris On Mon, Aug 30, 2010 at 8:40 PM, Janna Wemekamp <[email protected]> wrote: > I'm experimenting with the latest Fedora Commons 3.4 release and its Akubra > Low-Level Storage module. > I'm building the FOXML files using a custom Perl application which calls the > REST API's getNextPID to obtain the PID for each object. > After 754 calls, Java crashes! > Note - there were no other processes accessing this FC instance. > > The next getNextPID request returned: HTTP Response status: 503 Service > Temporarily Unavailable > > Fedora log: > > INFO 2010-08-31 09:56:29.683 [TP-Processor3] (DefaultManagement) Completed > getNextPID(numPIDs: 1, namespace: null) > INFO 2010-08-31 09:56:29.723 [TP-Processor3] (DefaultManagement) Completed > getNextPID(numPIDs: 1, namespace: null) > ERROR 2010-08-31 09:56:29.759 [TP-Processor3] (XmlUsersFileModule) > /opt/fedoracommons/nrm/3.4/dev/server/config/fedora-users.xml (Too many open > files) > > From the Tomcat log (catalina.out): > > # > # A fatal error has been detected by the Java Runtime Environment: > # > # Internal Error (exceptions.cpp:367), pid=25754, tid=1117292864 > # Error: ExceptionMark destructor expects no pending exceptions > # > # JRE version: 6.0_21-b06 > # Java VM: Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode > linux-amd64 ) > # Can not save log file, dump to screen.. > # > # A fatal error has been detected by the Java Runtime Environment: > # > # Internal Error (exceptions.cpp:367), pid=25754, tid=1117292864 > # Error: ExceptionMark destructor expects no pending exceptions > # > # JRE version: 6.0_21-b06 > # Java VM: Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode > linux-amd64 ) > # If you would like to submit a bug report, please visit: > # http://java.sun.com/webapps/bugreport/crash.jsp > # > > --------------- T H R E A D --------------- > > Current thread (0x00002aab3c00c000): JavaThread "TP-Processor3" daemon > [_thread_in_vm, id=25816, stack(0x0000000042888000,0x0000000042989000)] > > Stack: [0x0000000042888000,0x0000000042989000], sp=0x0000000042983e00, > free space=3ef0000000000000018k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > V [libjvm.so+0x730610] > V [libjvm.so+0x2f2ff6] > V [libjvm.so+0x342d3d] > V [libjvm.so+0x2870d8] > V [libjvm.so+0x286722] > V [libjvm.so+0x2867a6] > V [libjvm.so+0x2881b6] > V [libjvm.so+0x6c1abf] > V [libjvm.so+0x6bfc4b] > V [libjvm.so+0x6beae1] > V [libjvm.so+0x6beb10] > V [libjvm.so+0x45326b] > > Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) > j > java.lang.ClassLoader.findBootstrapClass(Ljava/lang/String;)Ljava/lang/Class;+0 > J java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; > J java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; > j > sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+41 > J java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; > j > java.util.ResourceBundle$RBClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+10 > j > java.util.ResourceBundle$Control.newBundle(Ljava/lang/String;Ljava/util/Locale;Ljava/lang/String;Ljava/lang/ClassLoader;Z)Ljava/util/ResourceBundle;+24 > j > java.util.ResourceBundle.loadBundle(Ljava/util/ResourceBundle$CacheKey;Ljava/util/List;Ljava/util/ResourceBundle$Control;Z)Ljava/util/ResourceBundle;+54 > j > java.util.ResourceBundle.findBundle(Ljava/util/ResourceBundle$CacheKey;Ljava/util/List;Ljava/util/List;ILjava/util/ResourceBundle$Control;Ljava/util/ResourceBundle;)Ljava/util/ResourceBundle;+213 > j > java.util.ResourceBundle.findBundle(Ljava/util/ResourceBundle$CacheKey;Ljava/util/List;Ljava/util/List;ILjava/util/ResourceBundle$Control;Ljava/util/ResourceBundle;)Ljava/util/ResourceBundle;+37 > j > java.util.ResourceBundle.getBundleImpl(Ljava/lang/String;Ljava/util/Locale;Ljava/lang/ClassLoader;Ljava/util/ResourceBundle$Control;)Ljava/util/ResourceBundle;+187 > j > java.util.ResourceBundle.getBundle(Ljava/lang/String;)Ljava/util/ResourceBundle;+10 > j sun.security.util.ResourcesMgr$1.run()Ljava/lang/Object;+2 > v ~StubRoutines::call_stub > J > java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; > j > sun.security.util.ResourcesMgr.getString(Ljava/lang/String;)Ljava/lang/String;+13 > j javax.security.auth.login.LoginContext.invoke(Ljava/lang/String;)V+1075 > j > javax.security.auth.login.LoginContext.access$000(Ljavax/security/auth/login/LoginContext;Ljava/lang/String;)V+2 > j javax.security.auth.login.LoginContext$4.run()Ljava/lang/Object;+8 > v ~StubRoutines::call_stub > j > java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;)Ljava/lang/Object;+0 > j javax.security.auth.login.LoginContext.invokePriv(Ljava/lang/String;)V+9 > j javax.security.auth.login.LoginContext.login()V+48 > j > org.fcrepo.server.security.jaas.AuthFilterJAAS.authenticate(Ljavax/servlet/http/HttpServletRequest;)Ljavax/security/auth/Subject;+197 > j > org.fcrepo.server.security.jaas.AuthFilterJAAS.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+290 > j > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+119 > j > org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+101 > j > org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+804 > j > org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+365 > j > org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+675 > j > org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+64 > j > org.apache.catalina.valves.ErrorReportValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+6 > j > org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+42 > j > org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V+157 > j > org.apache.jk.server.JkCoyoteHandler.invoke(Lorg/apache/jk/core/Msg;Lorg/apache/jk/core/MsgContext;)I+123 > j > org.apache.jk.common.HandlerRequest.invoke(Lorg/apache/jk/core/Msg;Lorg/apache/jk/core/MsgContext;)I+374 > j > org.apache.jk.common.ChannelSocket.invoke(Lorg/apache/jk/core/Msg;Lorg/apache/jk/core/MsgContext;)I+191 > j > org.apache.jk.common.ChannelSocket.processConnection(Lorg/apache/jk/core/MsgContext;)V+105 > j > org.apache.jk.common.ChannelSocket$SocketConnection.runIt([Ljava/lang/Object;)V+8 > j org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()V+167 > j java.lang.Thread.run()V+11 > v ~StubRoutines::call_stub > > --------------- P R O C E S S --------------- > > Java Threads: ( => current thread ) > 0x0000000049889800 JavaThread "pool-1-thread-3" [_thread_blocked, > id=26176, stack(0x0000000043292000,0x0000000043393000)] > 0x0000000049d2d800 JavaThread "TP-Processor12" daemon [_thread_in_native, > id=25907, stack(0x0000000043191000,0x0000000043292000)] > 0x00000000497ab000 JavaThread "TP-Processor11" daemon [_thread_in_native, > id=25906, stack(0x0000000043090000,0x0000000043191000)] > 0x00000000497aa800 JavaThread "TP-Processor10" daemon [_thread_blocked, > id=25905, stack(0x0000000042f8f000,0x0000000043090000)] > 0x000000004971d000 JavaThread "TP-Processor9" daemon [_thread_blocked, > id=25904, stack(0x00000000406d5000,0x00000000407d6000)] > 0x00000000497af800 JavaThread "TP-Processor8" daemon [_thread_in_native, > id=25836, stack(0x0000000042e8e000,0x0000000042f8f000)] > 0x00000000497ae800 JavaThread "TP-Processor7" daemon [_thread_in_native, > id=25835, stack(0x0000000042d8d000,0x0000000042e8e000)] > 0x000000004a22f800 JavaThread "TP-Processor6" daemon [_thread_in_native, > id=25834, stack(0x0000000042c8c000,0x0000000042d8d000)] > 0x000000004a0a6800 JavaThread "TP-Processor5" daemon [_thread_in_native, > id=25833, stack(0x0000000042b8b000,0x0000000042c8c000)] > 0x00002aab3c014000 JavaThread "TP-Monitor" daemon [_thread_blocked, > id=25818, stack(0x0000000042a8a000,0x0000000042b8b000)] > 0x00002aab3c012800 JavaThread "TP-Processor4" daemon [_thread_in_native, > id=25817, stack(0x0000000042989000,0x0000000042a8a000)] > =>0x00002aab3c00c000 JavaThread "TP-Processor3" daemon [_thread_in_vm, > id=25816, stack(0x0000000042888000,0x0000000042989000)] > 0x00002aab3c00b000 JavaThread "TP-Processor2" daemon [_thread_in_native, > id=25815, stack(0x0000000042787000,0x0000000042888000)] > 0x00002aab3c011000 JavaThread "TP-Processor1" daemon [_thread_in_native, > id=25814, stack(0x0000000042686000,0x0000000042787000)] > 0x00002aab3c006000 JavaThread "http-8080-Acceptor-0" daemon > [_thread_in_native, id=25813, stack(0x00000000403ff000,0x0000000040500000)] > 0x00002aab3c005800 JavaThread > "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon > [_thread_blocked, id=25812, stack(0x00000000402fe000,0x00000000403ff000)] > 0x0000000049b80800 JavaThread "pool-4-thread-1" [_thread_blocked, > id=25811, stack(0x00000000401fd000,0x00000000402fe000)] > 0x0000000049c52800 JavaThread "pool-2-thread-1" [_thread_blocked, > id=25810, stack(0x000000004095b000,0x0000000040a5c000)] > 0x00002aab310f6000 JavaThread "Thread-4" [_thread_blocked, id=25809, > stack(0x0000000042585000,0x0000000042686000)] > 0x00002aab3028c000 JavaThread "Thread-3" [_thread_blocked, id=25808, > stack(0x0000000042484000,0x0000000042585000)] > 0x00002aab3081d000 JavaThread "Write-lock Reaper" daemon [_thread_blocked, > id=25806, stack(0x000000004085a000,0x000000004095b000)] > 0x00002aab3125c800 JavaThread "TripleWriteThread: > /data/fedoracommons/nrm/3.4/dev/resourceIndex/xaStatementStore/xa.g_3201" > daemon [_thread_blocked, id=25805, > stack(0x0000000041518000,0x0000000041619000)] > 0x00002aab311b5800 JavaThread "TripleWriteThread: > /data/fedoracommons/nrm/3.4/dev/resourceIndex/xaStatementStore/xa.g_3120" > daemon [_thread_blocked, id=25804, > stack(0x0000000042383000,0x0000000042484000)] > 0x00002aab310fb000 JavaThread "TripleWriteThread: > /data/fedoracommons/nrm/3.4/dev/resourceIndex/xaStatementStore/xa.g_3012" > daemon [_thread_blocked, id=25803, > stack(0x0000000041417000,0x0000000041518000)] > 0x00002aab30626000 JavaThread "JotmClock" daemon [_thread_blocked, > id=25802, stack(0x00000000417f7000,0x00000000418f8000)] > 0x00002aab30393800 JavaThread "JotmBatch" daemon [_thread_blocked, > id=25801, stack(0x00000000416f6000,0x00000000417f7000)] > 0x00002aab30398800 JavaThread "GC Daemon" daemon [_thread_blocked, > id=25800, stack(0x00000000411f5000,0x00000000412f6000)] > 0x00002aab30941800 JavaThread "RMI Reaper" [_thread_blocked, id=25799, > stack(0x00000000410f4000,0x00000000411f5000)] > 0x00002aab310b2800 JavaThread "RMI TCP Accept-0" daemon > [_thread_in_native, id=25798, stack(0x0000000040ff3000,0x00000000410f4000)] > 0x0000000049f9a800 JavaThread "Thread-1" [_thread_blocked, id=25771, > stack(0x00000000419cb000,0x0000000041acc000)] > 0x0000000049611800 JavaThread "Low Memory Detector" daemon > [_thread_blocked, id=25766, stack(0x0000000042181000,0x0000000042282000)] > 0x000000004960f000 JavaThread "CompilerThread1" daemon [_thread_blocked, > id=25765, stack(0x0000000042080000,0x0000000042181000)] > 0x0000000049609800 JavaThread "CompilerThread0" daemon [_thread_blocked, > id=25764, stack(0x0000000041f7f000,0x0000000042080000)] > 0x0000000049607800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, > id=25763, stack(0x0000000041e7e000,0x0000000041f7f000)] > 0x00000000495e3000 JavaThread "Finalizer" daemon [_thread_blocked, > id=25762, stack(0x0000000040ef2000,0x0000000040ff3000)] > 0x00000000495e1000 JavaThread "Reference Handler" daemon [_thread_blocked, > id=25761, stack(0x0000000040d8f000,0x0000000040e90000)] > 0x000000004957f800 JavaThread "main" [_thread_in_native, id=25755, > stack(0x0000000040ad3000,0x0000000040bd4000)] > > Other Threads: > 0x00000000495dc800 VMThread [stack: 0x0000000040c8e000,0x0000000040d8f000] > [id=25760] > 0x000000004961c000 WatcherThread [stack: > 0x0000000042282000,0x0000000042383000] [id=25767] > > VM state:not at safepoint (normal execution) > > VM Mutex/Monitor currently owned by a thread: None > > Heap > PSYoungGen total 221120K, used 88016K [0x00002aab01f80000, > 0x00002aab112b0000, 0x00002aab29420000) > eden space 218752K, 39% used > [0x00002aab01f80000,0x00002aab07328308,0x00002aab0f520000) > from space 2368K, 99% used > [0x00002aab11060000,0x00002aab112ac078,0x00002aab112b0000) > to space 12480K, 0% used > [0x00002aab0fa50000,0x00002aab0fa50000,0x00002aab10680000) > PSOldGen total 80512K, used 22495K [0x00002aaab3620000, > 0x00002aaab84c0000, 0x00002aab01f80000) > object space 80512K, 27% used > [0x00002aaab3620000,0x00002aaab4c17e00,0x00002aaab84c0000) > PSPermGen total 64768K, used 64672K [0x00002aaaae220000, > 0x00002aaab2160000, 0x00002aaab3620000) > object space 64768K, 99% used > [0x00002aaaae220000,0x00002aaab2148300,0x00002aaab2160000) > > Dynamic libraries: > Can not get library information for pid = 25816 > > VM Arguments: > jvm_args: > -Djava.util.logging.config.file=/opt/fedoracommons/nrm/3.4/dev/tomcat/conf/logging.properties > -Dsolr.solr.home=/opt/fedoracommons/nrm/3.4/dev/solr > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > -Djava.endorsed.dirs=/opt/fedoracommons/nrm/3.4/dev/tomcat/endorsed > -Dcatalina.base=/opt/fedoracommons/nrm/3.4/dev/tomcat > -Dcatalina.home=/opt/fedoracommons/nrm/3.4/dev/tomcat > -Djava.io.tmpdir=/opt/fedoracommons/nrm/3.4/dev/tomcat/temp > java_command: org.apache.catalina.startup.Bootstrap start > Launcher Type: SUN_STANDARD > > Environment Variables: > JAVA_HOME=/usr/lib/jvm/jre > PATH=/opt/fedoracommons/nrm/3.4/dev/server/bin:/opt/fedoracommons/nrm/3.4/dev/client/bin:/opt/fedoracommons/nrm/3.4/dev/tomcat/bin:/opt/ant/bin:/usr/lib/jvm/jre/bin:/opt/openoffice.org3/program:/opt/ruby/1.8.7/bin:/opt/perl/5.10.0/bin:/opt/CollabNet_Subversion/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/fedora/bin > LD_LIBRARY_PATH=/usr/lib/jvm/java-1.6.0-sun-1.6.0.21.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.6.0-sun-1.6.0.21.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.6.0-sun-1.6.0.21.x86_64/jre/../lib/amd64 > SHELL=/bin/bash > DISPLAY=192.168.198.102:0 > > Signal Handlers: > SIGSEGV: [libjvm.so+0x7311c0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGBUS: [libjvm.so+0x7311c0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGFPE: [libjvm.so+0x5f5da0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGPIPE: [libjvm.so+0x5f5da0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGXFSZ: [libjvm.so+0x5f5da0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGILL: [libjvm.so+0x5f5da0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000 > SIGUSR2: [libjvm.so+0x5f85c0], sa_mask[0]=0x00000000, sa_flags=0x10000004 > SIGHUP: [libjvm.so+0x5f8310], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000 > SIGTERM: [libjvm.so+0x5f8310], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > SIGQUIT: [libjvm.so+0x5f8310], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 > > > --------------- S Y S T E M --------------- > > OS:Linux > uname:Linux 2.6.18-194.8.1.el5xen #1 SMP Wed Jun 23 11:01:41 EDT 2010 x86_64 > libc:glibc 2.5 NPTL 2.5 > rlimit: STACK 10240k, CORE 0k, NPROC 60404, NOFILE 1024, AS infinity > load average:0.00 0.00 0.00 > > CPU:total 4 (4 cores per cpu, 1 threads per core) family 6 model 23 stepping > 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1 > > Memory: 4k page, physical 7724032k(20328k free), swap 16771820k(16771496k > free) > > vm_info: Java HotSpot(TM) 64-Bit Server VM (17.0-b16) for linux-amd64 JRE > (1.6.0_21-b06), built on Jun 22 2010 01:10:00 by "java_re" with gcc 3.2.2 > (SuSE Linux) > > time: Tue Aug 31 09:56:29 2010 > elapsed time: 1381 seconds > > # > # If you would like to submit a bug report, please visit: > # http://java.sun.com/webapps/bugreport/crash.jsp > # > > Any ideas? > > TIA! > > > - Janna Wemekamp > - Toldark Pty Limited > > > > > > > > ------------------------------------------------------------------------------ > This SF.net Dev2Dev email is sponsored by: > > Show off your parallel programming skills. > Enter the Intel(R) Threading Challenge 2010. > http://p.sf.net/sfu/intel-thread-sfd > _______________________________________________ > Fedora-commons-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users > > ------------------------------------------------------------------------------ This SF.net Dev2Dev email is sponsored by: Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd _______________________________________________ Fedora-commons-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
