Another item of information: Ingesting FOXML files works without a problem, induicating that this may be a problem with the copy/replace functionality?
Also, I simplified the process somewhat: really, the new data is spread across three different filesystems, with the symlinks occuring at the top-level hash character directories: /fedora/data/objects/0 -> /fedora/db01/objects/0 /fedora/data/objects/1 -> /fedora/db02/objects/1 etc. I mention this, as it occurs to me that perhaps Akubra, as part of its safe move/rename function, is attempting to create temporary hard links across filesystems? Just a guess... -- Scott On 10/05/10, "Scott Prater" <pra...@wisc.edu> wrote: > We encountered a worrisome problem today after filling up a filesystem and > being forced to move our data to another filesystem. After the move and > rebuild, we can no longer add datastreams to objects. > > I'm running Fedora 3.4, with the default Akubra low-level storage module. > > Here's what happened: > > 1. filesystem /fedora/data filled up, Fedora (and Tomcat) hung. > 2. I shut down Fedora, Tomcat, Postgres. > 3. I moved the directory /fedora/data/objects to another filesystem, > /fedora/db01/objects. > 4. I moved the directory /fedora/data/datastreams to another filesystem, > /fedora/db01/datastreams. > 5. I made symlinks from the original locations to the new locations: > /fedora/data/objects -> /fedora/db01/objects > /fedora/data/datastreams -> /fedora/db01/datastreams > 6. I started up Postgres, and ran fedora-rebuild.sh twice, once to rebuild > the database and once to rebuild the resource index. The rebuild completed > successfully. > 7. I then started up Tomcat + Fedora, and I was able to view all the objects. > 8. I then did a test ingest of an object, and a stub object was successfully > created, with a DC datastream. I verified that I could find the object and > the datastream on the filesystem (both through the direct path and through > the symlinked path). I verified that permsissions were correct at every > directory level, and at the file level. > > I then try to add a datastream to the test object, and I receive this error > (truncated): > > ERROR 2010-10-05 14:51:16.424 [http-9090-1] (WebApplicationImpl) Internal > server error > javax.ws.rs.WebApplicationException: > org.fcrepo.server.errors.GeneralException: Unable to add or modify object > (commit canceled) > [...] > Caused by: java.io.IOException: Rename failed for an unknown reason. > at org.akubraproject.fs.FSBlob.moveTo(FSBlob.java:151) > [akubra-fs-0.3.jar:na] > > I've tested this with the REST API, through the web admin interface, and > through the java GUI admin tool, and the results are the same in all three > cases. This error occurs regardless of whether I'm adding a datastream to a > newly-created object or attempting to add a datastream to an older object. > > This may properly be an Akubra matter, but I thought I would bring it up > here, first. Any suggestions or help would be greatly appreciated. > > thanks, > > -- Scott > > Following is the full stack trace: > > ERROR 2010-10-05 14:51:16.424 [http-9090-1] (WebApplicationImpl) Internal > server error > javax.ws.rs.WebApplicationException: > org.fcrepo.server.errors.GeneralException: Unable to add or modify object > (commit canceled) > at > org.fcrepo.server.rest.BaseRestResource.handleException(BaseRestResource.java:168) > [fcrepo-server-3.4.jar:na] > at > org.fcrepo.server.rest.DatastreamResource.addOrUpdateDatastream(DatastreamResource.java:583) > [fcrepo-server-3.4.jar:na] > at > org.fcrepo.server.rest.DatastreamResource.addDatastream(DatastreamResource.java:358) > [fcrepo-server-3.4.jar:na] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [na:1.6.0_21] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > [na:1.6.0_21] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > [na:1.6.0_21] > at java.lang.reflect.Method.invoke(Method.java:597) [na:1.6.0_21] > at > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:175) > [jersey-bundle-1.0.3.1.jar:1.0.3.1] > at > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67) > [jersey-bundle-1.0.3.1.jar:1.0.3.1] > at > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:163) > [jersey-bundle-1.0.3.1.jar:1.0.3.1] > at > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:111) > [jersey-bundle-1.0.3.1.jar:1.0.3.1] > at > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:71) > [jersey-bundle-1.0.3.1.jar:1.0.3.1] > at > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:111) > [jersey-bundle-1.0.3.1.jar:1.0.3.1] > at > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:63) > [jersey-bundle-1.0.3.1.jar:1.0.3.1] > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:689) > [jersey-bundle-1.0.3.1.jar:1.0.3.1] > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:647) > [jersey-bundle-1.0.3.1.jar:1.0.3.1] > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:638) > [jersey-bundle-1.0.3.1.jar:1.0.3.1] > at > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:309) > [jersey-bundle-1.0.3.1.jar:1.0.3.1] > at > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:425) > [jersey-bundle-1.0.3.1.jar:1.0.3.1] > at > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:590) > [jersey-bundle-1.0.3.1.jar:1.0.3.1] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > [servlet-api.jar:na] > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > [catalina.jar:6.0.29] > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > [catalina.jar:6.0.29] > at > org.fcrepo.server.security.servletfilters.FilterRestApiFlash.doFilter(FilterRestApiFlash.java:66) > [fcrepo-server-3.4.jar:na] > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > [catalina.jar:6.0.29] > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > [catalina.jar:6.0.29] > at > org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235) > [fcrepo-server-3.4.jar:na] > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > [catalina.jar:6.0.29] > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > [catalina.jar:6.0.29] > at > org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235) > [fcrepo-server-3.4.jar:na] > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > [catalina.jar:6.0.29] > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > [catalina.jar:6.0.29] > at > org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235) > [fcrepo-server-3.4.jar:na] > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > [catalina.jar:6.0.29] > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > [catalina.jar:6.0.29] > at > org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235) > [fcrepo-server-3.4.jar:na] > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > [catalina.jar:6.0.29] > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > [catalina.jar:6.0.29] > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > [catalina.jar:6.0.29] > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > [catalina.jar:6.0.29] > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563) > [catalina.jar:6.0.29] > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > [catalina.jar:6.0.29] > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > [catalina.jar:6.0.29] > at > org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269) > [catalina.jar:6.0.29] > at > org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81) > [catalina.jar:6.0.29] > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) > [catalina.jar:6.0.29] > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > [catalina.jar:6.0.29] > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > [catalina.jar:6.0.29] > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) > [tomcat-coyote.jar:6.0.29] > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) > [tomcat-coyote.jar:6.0.29] > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > [tomcat-coyote.jar:6.0.29] > at java.lang.Thread.run(Thread.java:619) [na:1.6.0_21] > Caused by: org.fcrepo.server.errors.GeneralException: Unable to add or modify > object (commit canceled) > at > org.fcrepo.server.storage.DefaultDOManager.doCommit(DefaultDOManager.java:1438) > [fcrepo-server-3.4.jar:na] > at > org.fcrepo.server.storage.SimpleDOWriter.commit(SimpleDOWriter.java:508) > [fcrepo-server-3.4.jar:na] > at > org.fcrepo.server.management.DefaultManagement.addDatastream(DefaultManagement.java:572) > [fcrepo-server-3.4.jar:na] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [na:1.6.0_21] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > [na:1.6.0_21] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > [na:1.6.0_21] > at java.lang.reflect.Method.invoke(Method.java:597) [na:1.6.0_21] > at > org.fcrepo.server.messaging.NotificationInvocationHandler.invoke(NotificationInvocationHandler.java:68) > [fcrepo-server-3.4.jar:na] > at $Proxy0.addDatastream(Unknown Source) [na:na] > at > org.fcrepo.server.management.ManagementModule.addDatastream(ManagementModule.java:227) > [fcrepo-server-3.4.jar:na] > at > org.fcrepo.server.rest.DatastreamResource.addOrUpdateDatastream(DatastreamResource.java:467) > [fcrepo-server-3.4.jar:na] > ... 50 common frames omitted > Caused by: org.fcrepo.common.FaultException: java.io.IOException: Rename > failed for an unknown reason. > at > org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.rename(AkubraLowlevelStorage.java:363) > [fcrepo-server-3.4.jar:na] > at > org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.safeOverwrite(AkubraLowlevelStorage.java:306) > [fcrepo-server-3.4.jar:na] > at > org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.replace(AkubraLowlevelStorage.java:244) > [fcrepo-server-3.4.jar:na] > at > org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.replaceObject(AkubraLowlevelStorage.java:150) > [fcrepo-server-3.4.jar:na] > at > org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorageModule.replaceObject(AkubraLowlevelStorageModule.java:79) > [fcrepo-server-3.4.jar:na] > at > org.fcrepo.server.storage.DefaultDOManager.doCommit(DefaultDOManager.java:1334) > [fcrepo-server-3.4.jar:na] > ... 60 common frames omitted > Caused by: java.io.IOException: Rename failed for an unknown reason. > at org.akubraproject.fs.FSBlob.moveTo(FSBlob.java:151) > [akubra-fs-0.3.jar:na] > at org.akubraproject.map.IdMappingBlob.moveTo(IdMappingBlob.java:69) > [akubra-map-0.3.jar:na] > at > org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.rename(AkubraLowlevelStorage.java:361) > [fcrepo-server-3.4.jar:na] > ... 65 common frames omitted > -- > Scott Prater > Library, Instructional, and Research Applications (LIRA) > Division of Information Technology (DoIT) > University of Wisconsin - Madison > pra...@wisc.edu ------------------------------------------------------------------------------ Beautiful is writing same markup. Internet Explorer 9 supports standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3. Spend less time writing and rewriting code and more time creating great experiences on the web. Be a part of the beta today. http://p.sf.net/sfu/beautyoftheweb _______________________________________________ Fedora-commons-users mailing list Fedora-commons-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
