I added 1 GB of DRAM to my server (latest 1.3.x) and that dramatically
improved the slow response times I was experiencing (due to swapping, no
doubt) and the "Out of memory" errors I was seeing pretty much went away.
Swapping is not unlike having a heavily loaded system and if there are
conditions under which I'm getting "Out of memory" then I can pretty much
guarantee that other users will experience the same situation under some
kind of peak load condition when the server becomes pretty busy doing
stuff. I don't know how to prevent that and test for the combinations which
will line up to cause this out of memory condition. Obviously, increasing
memory limits will cure it, but that seems like a solution where you have
to hold your breath and hope it doesn't happen very often. Would be nice if
there was some way to assess max memory usage other than empirical
observations.
With my new extra DRAM I uploaded eight short videos and they all ingested
fine. But then I tried to retract all eight at once, and got Java heap
errors (see below). I don't know who meters and controls resource usage,
and whether retract needs to check how many workflows are "Initializing..."
or pending or whatever. The system seems to limit workflow processing to
two jobs at once, probably one for each core on my test server, but it
looks like pending work is also eating up heap space even though not active.
I don't know what gatekeepers are missing or needed, but it's pretty clear
that there is another type of race condition here that's going to cause
real failures when the system gets busy.
Performance like this shouldn't be left to chance. Process flow should be
deterministic and guaranteed to run.
The fix is obvious (increase heap space), but thought I'd pass along these
observations.
Regards,
Hank
19:08:46 ERROR (WorkflowOperationWorker:157) - Workflow operation
'retract-download' failed
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133)
at java.lang.StringCoding.decode(StringCoding.java:173)
at java.lang.String.<init>(String.java:443)
at org.postgresql.core.Encoding.decode(Encoding.java:193)
at org.postgresql.core.Encoding.decode(Encoding.java:205)
at
org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(AbstractJdbc2ResultSet.java:1879)
at
com.mchange.v2.c3p0.impl.NewProxyResultSet.getString(NewProxyResultSet.java:3316)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.getObjectThroughOptimizedDataConversion(DatabaseAccessor.java:1189)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.getObject(DatabaseAccessor.java:1100)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.fetchRow(DatabaseAccessor.java:930)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:653)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:530)
at
org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:529)
at
org.eclipse.persistence.internal.sessions.IsolatedClientSession.executeCall(IsolatedClientSession.java:133)
at
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:206)
at
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:192)
at
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:263)
at
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelect(DatasourceCallQueryMechanism.java:245)
at
org.eclipse.persistence.queries.DataReadQuery.executeNonCursor(DataReadQuery.java:188)
at
org.eclipse.persistence.queries.DataReadQuery.executeDatabaseQuery(DataReadQuery.java:144)
at
org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:675)
at
org.eclipse.persistence.queries.DataReadQuery.execute(DataReadQuery.java:130)
at
org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:589)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2898)
at
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1225)
at
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1207)
at
org.eclipse.persistence.internal.indirection.NoIndirectionPolicy.valueFromQuery(NoIndirectionPolicy.java:299)
at
org.eclipse.persistence.mappings.DirectCollectionMapping.valueFromRow(DirectCollectionMapping.java:2834)
at
org.eclipse.persistence.mappings.ForeignReferenceMapping.buildCloneFromRow(ForeignReferenceMapping.java:195)
at
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoWorkingCopyClone(ObjectBuilder.java:1261)
at
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:1388)
at
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:548)
19:08:51 ERROR (WorkflowOperationWorker:157) - Workflow operation
'retract-download' failed
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133)
at java.lang.StringCoding.decode(StringCoding.java:173)
at java.lang.String.<init>(String.java:443)
at org.postgresql.core.Encoding.decode(Encoding.java:193)
at org.postgresql.core.Encoding.decode(Encoding.java:205)
at
org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(AbstractJdbc2ResultSet.java:1879)
_______________________________________________
Matterhorn-users mailing list
[email protected]
http://lists.opencastproject.org/mailman/listinfo/matterhorn-users
