We encountered a worrisome problem today after filling up a filesystem and
being forced to move our data to another filesystem. After the move and
rebuild, we can no longer add datastreams to objects.
I'm running Fedora 3.4, with the default Akubra low-level storage module.
Here's what happened:
1. filesystem /fedora/data filled up, Fedora (and Tomcat) hung.
2. I shut down Fedora, Tomcat, Postgres.
3. I moved the directory /fedora/data/objects to another filesystem,
/fedora/db01/objects.
4. I moved the directory /fedora/data/datastreams to another filesystem,
/fedora/db01/datastreams.
5. I made symlinks from the original locations to the new locations:
/fedora/data/objects -> /fedora/db01/objects
/fedora/data/datastreams -> /fedora/db01/datastreams
6. I started up Postgres, and ran fedora-rebuild.sh twice, once to rebuild the
database and once to rebuild the resource index. The rebuild completed
successfully.
7. I then started up Tomcat + Fedora, and I was able to view all the objects.
8. I then did a test ingest of an object, and a stub object was successfully
created, with a DC datastream. I verified that I could find the object and the
datastream on the filesystem (both through the direct path and through the
symlinked path). I verified that permsissions were correct at every directory
level, and at the file level.
I then try to add a datastream to the test object, and I receive this error
(truncated):
ERROR 2010-10-05 14:51:16.424 [http-9090-1] (WebApplicationImpl) Internal
server error
javax.ws.rs.WebApplicationException: org.fcrepo.server.errors.GeneralException:
Unable to add or modify object (commit canceled)
[...]
Caused by: java.io.IOException: Rename failed for an unknown reason.
at org.akubraproject.fs.FSBlob.moveTo(FSBlob.java:151)
[akubra-fs-0.3.jar:na]
I've tested this with the REST API, through the web admin interface, and
through the java GUI admin tool, and the results are the same in all three
cases. This error occurs regardless of whether I'm adding a datastream to a
newly-created object or attempting to add a datastream to an older object.
This may properly be an Akubra matter, but I thought I would bring it up here,
first. Any suggestions or help would be greatly appreciated.
thanks,
-- Scott
Following is the full stack trace:
ERROR 2010-10-05 14:51:16.424 [http-9090-1] (WebApplicationImpl) Internal
server error
javax.ws.rs.WebApplicationException: org.fcrepo.server.errors.GeneralException:
Unable to add or modify object (commit canceled)
at
org.fcrepo.server.rest.BaseRestResource.handleException(BaseRestResource.java:168)
[fcrepo-server-3.4.jar:na]
at
org.fcrepo.server.rest.DatastreamResource.addOrUpdateDatastream(DatastreamResource.java:583)
[fcrepo-server-3.4.jar:na]
at
org.fcrepo.server.rest.DatastreamResource.addDatastream(DatastreamResource.java:358)
[fcrepo-server-3.4.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[na:1.6.0_21]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[na:1.6.0_21]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[na:1.6.0_21]
at java.lang.reflect.Method.invoke(Method.java:597) [na:1.6.0_21]
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:175)
[jersey-bundle-1.0.3.1.jar:1.0.3.1]
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67)
[jersey-bundle-1.0.3.1.jar:1.0.3.1]
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:163)
[jersey-bundle-1.0.3.1.jar:1.0.3.1]
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:111)
[jersey-bundle-1.0.3.1.jar:1.0.3.1]
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:71)
[jersey-bundle-1.0.3.1.jar:1.0.3.1]
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:111)
[jersey-bundle-1.0.3.1.jar:1.0.3.1]
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:63)
[jersey-bundle-1.0.3.1.jar:1.0.3.1]
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:689)
[jersey-bundle-1.0.3.1.jar:1.0.3.1]
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:647)
[jersey-bundle-1.0.3.1.jar:1.0.3.1]
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:638)
[jersey-bundle-1.0.3.1.jar:1.0.3.1]
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:309)
[jersey-bundle-1.0.3.1.jar:1.0.3.1]
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:425)
[jersey-bundle-1.0.3.1.jar:1.0.3.1]
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:590)
[jersey-bundle-1.0.3.1.jar:1.0.3.1]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
[servlet-api.jar:na]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
[catalina.jar:6.0.29]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
[catalina.jar:6.0.29]
at
org.fcrepo.server.security.servletfilters.FilterRestApiFlash.doFilter(FilterRestApiFlash.java:66)
[fcrepo-server-3.4.jar:na]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
[catalina.jar:6.0.29]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
[catalina.jar:6.0.29]
at
org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235)
[fcrepo-server-3.4.jar:na]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
[catalina.jar:6.0.29]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
[catalina.jar:6.0.29]
at
org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235)
[fcrepo-server-3.4.jar:na]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
[catalina.jar:6.0.29]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
[catalina.jar:6.0.29]
at
org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235)
[fcrepo-server-3.4.jar:na]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
[catalina.jar:6.0.29]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
[catalina.jar:6.0.29]
at
org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235)
[fcrepo-server-3.4.jar:na]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
[catalina.jar:6.0.29]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
[catalina.jar:6.0.29]
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
[catalina.jar:6.0.29]
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
[catalina.jar:6.0.29]
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
[catalina.jar:6.0.29]
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
[catalina.jar:6.0.29]
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
[catalina.jar:6.0.29]
at
org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269)
[catalina.jar:6.0.29]
at
org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81)
[catalina.jar:6.0.29]
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
[catalina.jar:6.0.29]
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
[catalina.jar:6.0.29]
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
[catalina.jar:6.0.29]
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
[tomcat-coyote.jar:6.0.29]
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
[tomcat-coyote.jar:6.0.29]
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
[tomcat-coyote.jar:6.0.29]
at java.lang.Thread.run(Thread.java:619) [na:1.6.0_21]
Caused by: org.fcrepo.server.errors.GeneralException: Unable to add or modify
object (commit canceled)
at
org.fcrepo.server.storage.DefaultDOManager.doCommit(DefaultDOManager.java:1438)
[fcrepo-server-3.4.jar:na]
at
org.fcrepo.server.storage.SimpleDOWriter.commit(SimpleDOWriter.java:508)
[fcrepo-server-3.4.jar:na]
at
org.fcrepo.server.management.DefaultManagement.addDatastream(DefaultManagement.java:572)
[fcrepo-server-3.4.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[na:1.6.0_21]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[na:1.6.0_21]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[na:1.6.0_21]
at java.lang.reflect.Method.invoke(Method.java:597) [na:1.6.0_21]
at
org.fcrepo.server.messaging.NotificationInvocationHandler.invoke(NotificationInvocationHandler.java:68)
[fcrepo-server-3.4.jar:na]
at $Proxy0.addDatastream(Unknown Source) [na:na]
at
org.fcrepo.server.management.ManagementModule.addDatastream(ManagementModule.java:227)
[fcrepo-server-3.4.jar:na]
at
org.fcrepo.server.rest.DatastreamResource.addOrUpdateDatastream(DatastreamResource.java:467)
[fcrepo-server-3.4.jar:na]
... 50 common frames omitted
Caused by: org.fcrepo.common.FaultException: java.io.IOException: Rename failed
for an unknown reason.
at
org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.rename(AkubraLowlevelStorage.java:363)
[fcrepo-server-3.4.jar:na]
at
org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.safeOverwrite(AkubraLowlevelStorage.java:306)
[fcrepo-server-3.4.jar:na]
at
org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.replace(AkubraLowlevelStorage.java:244)
[fcrepo-server-3.4.jar:na]
at
org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.replaceObject(AkubraLowlevelStorage.java:150)
[fcrepo-server-3.4.jar:na]
at
org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorageModule.replaceObject(AkubraLowlevelStorageModule.java:79)
[fcrepo-server-3.4.jar:na]
at
org.fcrepo.server.storage.DefaultDOManager.doCommit(DefaultDOManager.java:1334)
[fcrepo-server-3.4.jar:na]
... 60 common frames omitted
Caused by: java.io.IOException: Rename failed for an unknown reason.
at org.akubraproject.fs.FSBlob.moveTo(FSBlob.java:151)
[akubra-fs-0.3.jar:na]
at org.akubraproject.map.IdMappingBlob.moveTo(IdMappingBlob.java:69)
[akubra-map-0.3.jar:na]
at
org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.rename(AkubraLowlevelStorage.java:361)
[fcrepo-server-3.4.jar:na]
... 65 common frames omitted
--
Scott Prater
Library, Instructional, and Research Applications (LIRA)
Division of Information Technology (DoIT)
University of Wisconsin - Madison
pra...@wisc.edu
------------------------------------------------------------------------------
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
