Thanks for sharing that info Eric,
for the pooling exception it was a matter of doing session.close() after
obtaining the node or using the session ,
since there was no transaction the sessions where never closed nor reused.
I just tried the backup feature:
OutputStream output = new
FileOutputStream(rutaXmlBackup+"/jackRabbitBackup.xml");
System.out.println("Iniciando backup en '" +
rutaXmlBackup + "'");
session.exportSystemView("/", output, false, false);
it works great, 1400+ files thrown into an xml file in about 2 seconds.
Now i tried to restore it and the files are there but my "handmade"
repository viewer/browser seems to not see them.
Investigating it right now, might be that some duplicated stuff has been
created? Same name sibling i mean.
I expect to find a node let's say 'ROOT_NODE' , maybe all the restored
stuff is hanging off 'ROOT_NODE[1]' ...
Why do you think it's related to CPU Load? What connection do you see
between the files and the cpu load?
On 11/01/18 11:37, Eric Berryman wrote:
I have this exact problem. Sometimes, after a restart, I can find the
nodes, but it says it's empty ... but, I see the blobs of those attachments
in database, so they are "there".
For me, it seemed to correlate with a high CPU load, this might be related,
it might not.
I'm not much help, but I wanted to express equal concern on the matter.
Thank you!
Eric
On Thu, Jan 4, 2018 at 1:02 PM, Manuel López Blasi <
lopezbl...@conicet.gov.ar> wrote:
Thanks for the info Pontus.
It seems the second non-transactional pool fixed the connections leak, no
new exceptions
since yesterday so far. I gues the bindToTransaction was causing it.
I noticed something strange today: i left the server running since
yesterday night
until today morning, we saved many documents in the repository, no
problems so far.
Browsed the Repo with no issues, all documents are there, everything is
fine.
After a while we couldn't retrieve the documents from repository.
Let's say the tree is something like this:
rootNode
-> App1
->someCategory1
->Year1
->Month1
->Item1
->Year2
->someCategory2
-> App2
the log says PAthNotFoundException: Path 'App1' not found.This happens in
this code:
Node nodoRaiz = session.getRootNode();
Node nodoApp1 = nodoRaiz.getNode("App1");
i then restart Glassfish, now i can find 'App1' node , but not it's child
nodes. And i know
they're there. It's like the link between them got corrupted or something.
Any experience with possible data corruption?
Regards,
Manuel.
On 04/01/18 08:11, Pontus Amberg wrote:
I'm not sure but on some places it is stated that by default Glassfish has
not transaction timeout for
EJB:s. If that still is true then maybe you could try to set a transaction
timeout for the EJB that is
fetching the files to see if
1. The Jackrabbit session/connection is closed when the transaction
timeouts
2. Try to figure out why the transaction in your EJB isn't ended correctly
I see that you use
@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) and normally
I use TransactionAttributeType.REQUIRED in EJB:s. This will make sure
that
the method/methods are
included in a transaction and that a new transaction isn't started for
each
method call (if you call the
methods from other EJB:s or using the sessionContext.getBusinessObject()
to
invoke local methods).
But there's of course some cases where you actually want to use
TransactionAttributeType.REQUIRES_NEW
and this is maybe the case with your class/method.
/Pontus
On Wed, Jan 3, 2018 at 11:19 PM, Manuel López Blasi <
lopezbl...@conicet.gov.ar> wrote:
I have set transaction support to "Xa-Transaction" (In glassfish
properties) ,
i believe what i'm using is CMT, container managed transactions,
so glassfish is in charge of handling connections and kill them,
invalidate them or mark them as available
once the transaction has commited.
I discovered something interesting: when saving files everything went
smooth,
save thousands with no problem, seeing in glassfish monitoring console
how
many hundred
of connections are created, used, and disposed within seconds. Absolutely
no errors.
But when i get stuff from the repository, that's when connections begin
to
get leaked.
That's make me think that since there's no transaction commited the
connection stays open.
I tried to retrieve files from jackrabbit outside of a transactional
context and inside too, with no luck, same
PoolingException / out of connections error.
My service to reach jackrabbit, the place where the transactional context
begins are annotated like this:
@LocalBean
@Stateless
@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
I smell that's precisely the problem: since i have
"bindSessionToTransaction=true",
if i try to reach repository inside or outside any transaction, doesn't
matter, since
there's really no commit at all, connections will stale.
I'm going to try doing a second connection pool with
"bindSessionToTransaction=false",
using it only to retrieve stuff from jackrabbit.
I'll tell you how it goes soon, thanks a lot for your responses,
cheerz,
Manuel.
On 03/01/18 17:22, Pontus Amberg wrote:
What are you using to handle the transactions when you invoke the
Jackrabbit JCA connector? The reason I'm asking is that the flag
"bindSessionToTransaction=true" might maybe be an indication that you
have
transactions that for some reason never are committed.
/Pontus
On Tue, Jan 2, 2018 at 8:18 PM, Manuel López Blasi <
lopezbl...@conicet.gov.ar> wrote:
Monitoring Glassfish shows all connections are taken up ( and not freed
):
NumConnUsed 32count Jan 2, 2018 10:48:22 AM Jan 2,
2018 3:58:20 PM Marca de Agua Máxima: 32 count
Marca de Agua Mínima: 0 count
Provides connection usage statistics. The total number of
connections that are currently being used, as well as information about
the
maximum number of connections that were used (the high water mark).
All 32 connections are taken already.
On 02/01/18 15:00, Manuel López Blasi wrote:
Hello, thanks for your response Pontus,
i have set a maximun of concurrent connections, 32.
I understand that i set a maximum number o
sessions/connections/transactions,
in my case on glassfish.
These is handled by the jca connector y conjunction with the glassfish
server/container.
Once this maximun is reached, should i ask for another new connection,
the connector/connection pool would wait
until one of the bussy connections is freed. There is a wait timeout
for
this, once the time is elapsed the connection pool
would return an error message, saying that no connection is
available.
It's perfectly logical.
In my case this is happening, i get an exception "Connections in use
are
equal to max-pool-size value and max-wait-time has elapsed":
Caused by: com.sun.appserv.connectors.internal.api.PoolingException:
Las
conexiones en uso equivalen al valor de max-pool-size y el tiempo
caducado
de max-wait-time. No se pueden asignar m?s conexiones.
at com.sun.enterprise.resource.pool.ConnectionPool.getResource(
ConnectionPool.java:418)
at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource
FromPool(PoolManagerImpl.java:245)
at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource
(PoolManagerImpl.java:170)
at com.sun.enterprise.connectors.ConnectionManagerImpl.getResou
rce(ConnectionManagerImpl.java:332)
at com.sun.enterprise.connectors.ConnectionManagerImpl.internal
GetConnection(ConnectionManagerImpl.java:301)
at com.|#]
[#|2018-01-02T14:23:20.456-0300|SEVERE|glassfish3.1.2|javax.
enterprise.system.std.com.sun.enterprise.server.logging|_
ThreadID=409;_ThreadName=Thread-2;|sun.enterprise.
connectors.ConnectionManagerImpl.allocateConnection(Connecti
onManagerImpl.java:190)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocate
Connection(ConnectionManagerImpl.java:165)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocate
Connection(ConnectionManagerImpl.java:160)
at org.apache.jackrabbit.jca.JCARepositoryHandle.login(JCARepos
itoryHandle.java:75)
The thing is, once i reach this state it remains the same, i can wait
10
minutes or 5 hours, when it dies, it stays that way no matter how
long i
leave it "to recover connections".
The only solution is to shut down server and start it again, that way
everything works great again.
One other thing that seems strange is the fact that i can work
generating
files in number of thousands in very short time, let's say 2000 files
in 3
minutes. That may indicate
that the time settings for the connection pool are okay, i mean, i
have
1
minute of max wait time before saying there're no more free
connections,
almost a thousand files can be fully
processed and saved within 1 minute.
That's what leaves me perplexed. The other thing is that mysql
connection
pools have the very same / carbon copy settings and they work ok,
never
run
off connections or died this way.
I know files are way different, requires more work than db registers,
I/O
is the most time consuming and slow op of them all. Maybe within a
certain
amount of time the file caching gets
bottlenecked and that's what causes the collapse?
On 29/12/17 11:17, Pontus Amberg wrote:
Have you verified that it isn't the number of concurrent
sessions/transactions that is causing the problem? If that is the
problem
you would probably only encounter it when you have approximately 33
or
more
file operations executing at the same time.
/Pontus
On Tue, Dec 26, 2017 at 11:28 PM, Manuel López Blasi <
lopezbl...@conicet.gov.ar> wrote:
Hello,
i've been adding almost successfully Jackrabbit Repository to our
project,
basically for file storing purposes. Everything works great, with
some
exceptions,
one which is critical, once in a while, following no apparent
pattern
an
exception is thrown
saying the pool is out of connections, this one:
Caused by: com.sun.appserv.connectors.int
ernal.api.PoolingException:
Las conexiones en uso equivalen al valor de max-pool-size y el
tiempo
caducado de max-wait-time. No se pueden asignar m?s conexiones.
(Quantity of connections in use are same as defined max-pool-size
and
max-wait-time already elapsed. Can't assign any more connections.)
at com.sun.enterprise.resource.po
ol.ConnectionPool.getResource(
ConnectionPool.java:418)
at com.sun.enterprise.resource.po
ol.PoolManagerImpl.getResource
FromPool(PoolManagerImpl.java:245)
at com.sun.enterprise.resource.po
ol.PoolManagerImpl.getResource
(PoolManagerImpl.java:170)
at com.sun.enterprise.connectors.
ConnectionManagerImpl.getResou
rce(ConnectionManagerImpl.java:332)
at com.sun.enterprise.connectors.
ConnectionManagerImpl.internal
GetConnection(ConnectionManagerImpl.java:301)
at com.sun.enterprise.connectors.
ConnectionManagerImpl.allocate
Connection(ConnectionManagerImpl.java:190)
at com.sun.enterprise.connectors.
ConnectionManagerImpl.allocate
Connection(ConnectionManagerImpl.java:165)
at com.sun.enterprise.connectors.
ConnectionManagerImpl.allocate
Connection(ConnectionManagerImpl.java:160)
at org.apache.jackrabbit.jca.JCAR
epositoryHandle.login(JCARepos
itoryHandle.java:75)
... 120 more
Our setup/context is the following:
VM: java 7 (1.7.0_101)
container: Glassfish 3.1.2.2
main framework for webapp: struts 2
DB (mysql) persistence manager: Hibernate 4.2.19.Final
Jackrabbit stuff/versions:
jackrabbit-core 2.14.4
jcr 2.0
OCM: jackrabbit-ocm 2.0.0
Connector: jackrabbit-jca-2.14.4 (This one is deployed as a
connector
in
glassfish, associated with a connection pool )
The configuration for JCA connector is the following:
Connection definition: javax.jcr.Repository
Initial and minimum pool size: 8 Connections
Maximum pool size: 32 Connections
Switch Pool size: 2 connections
Activity Timeout 300 seconds
Max Wait Timeout: 60000 miliseconds
Transaction Support: XATransaction
Matching Connections: Yes.
bindSessionToTransaction: True
It seems to be caused randomnly, as we're able to produce and store
a
couple thousand of files within minutes with no crashes
(every file is stored within a transaction and with a single Session
to
the repository). Should the pool be out of connections,
it should happen immediately i think (???).
So, if someone has any indication/clues it would be greatly
appreciated,
thanks in advance, best regards,
Manuel.
