Re: [Neo] Online backup fails with suggested Spring transaction integration
Johan, Understood. In that case I suggest a simple Configuration interface with getConfig(), implemented by EmbeddedNeoService. This would suffice as a more appropriate solution to the configuration workaround IMO because then you only have to pass that interface to the transaction manager. Certainly, you had to extract the inner class Configuration then, too ;) Regardless, I am happy to wait until the next release for a proper solution and in the meantime can live with my own hacked version. Thanks for clarifying. -Andreas Johan Svensson wrote: Andreas, I applied most of it, both SpringTransactionManager and UserTransactionImpl now have the get/set method for transaction manager and the default constructor. I did not add the new constructor in EmbeddedNeo because it didn't feel right to have a constructor taking SpringTransactionManager there (NeoService knows nothing about Spring). We will add an EmbeddedNeo constructor that can take a javax.transaction.TransactionManager instead (probably through configuration map). -Johan On Thu, May 28, 2009 at 10:07 PM, Andreas Guenther aguent...@flying-orange.com wrote: Johan, What is the reason that you didn't apply the SpringTransactionManager injection changes to SpringTransactionManager, UserTransactionImpl, EmbeddedNeo, and NeoJvmInstance? -Andreas ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Online backup fails with suggested Spring transaction integration
Andreas, I applied most of it, both SpringTransactionManager and UserTransactionImpl now have the get/set method for transaction manager and the default constructor. I did not add the new constructor in EmbeddedNeo because it didn't feel right to have a constructor taking SpringTransactionManager there (NeoService knows nothing about Spring). We will add an EmbeddedNeo constructor that can take a javax.transaction.TransactionManager instead (probably through configuration map). -Johan On Thu, May 28, 2009 at 10:07 PM, Andreas Guenther aguent...@flying-orange.com wrote: Johan, What is the reason that you didn't apply the SpringTransactionManager injection changes to SpringTransactionManager, UserTransactionImpl, EmbeddedNeo, and NeoJvmInstance? -Andreas ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Online backup fails with suggested Spring transaction integration
Andreas,
I had a look at this and realized problem is caused by the static
nature of SpringTransactionManager and UserTransactionImpl. A fix for
this has been on my todo for some time but to make it right we need to
expose some new stuff in NeoService (mainly tied to configuration API
that has yet to be specified). For now I did a quick fix in
b9-SNAPSHOT changing the SpringTransactionManager
UserTransactionImpl classes to take NeoService in constructor.
Previously those two classes were singletons and the startup of a new
NeoService did just set a static field in them (not right). This
caused the backup slave's TM to be configured as the Spring TM and
that would in turn cause declarative transactions to fail.
I had to change your NeoServiceWrapper to take NeoService instead of a
string path to the neo db. This is because the NeoService you are
running with must be the same NeoService that was used to configure
and setup the Spring transaction manager.
I committed those changes to the test transaction project in your laboratory.
NOTE: The old suggested spring configuration (found on the wiki, use
it with neo-b8) will not work anymore when running neo-b9-SNAPSHOT.
Instead one has to do something like:
bean id=neo class=org.neo4j.api.core.EmbeddedNeo
destroy-method=shutdown lazy-init=false
constructor-arg
value${neo.store}/value
/constructor-arg
/bean
tx:annotation-driven transaction-manager=transactionManager/
alias name=neoTransactionManager alias=transactionManager/
bean id=neoTransactionManagerService
class=org.neo4j.impl.transaction.SpringTransactionManager
constructor-arg ref=neo/
/bean
bean id=neoUserTransactionService
class=org.neo4j.impl.transaction.UserTransactionImpl
constructor-arg ref=neo/
/bean
bean id=neoTransactionManager
class=org.springframework.transaction.jta.JtaTransactionManager
property name=transactionManager ref=neoTransactionManagerService/
property name=userTransaction ref=neoUserTransactionService/
/bean
Regards,
Johan
On Wed, May 20, 2009 at 2:00 AM, Andreas Guenther
aguent...@flying-orange.com wrote:
Johan, Emil,
Oki, I committed a test project resembling a simplification of what I am
using in our application.
https://svn.neo4j.org/laboratory/users/aguenther/testing/transaction
The two relevant test cases I am referring to are:
#1 ExplicitTransactionBackupTest (succeeds)
#2 SpringBasedTransactionBackupTest (fails)
The second test case relies on Spring's declarative transactions which
successfully work otherwise in our application.
I hope the code makes enough sense to spot the underlying problem. Ideally
it's just me not properly using Neo ;)
Thanks in advance,
-Andreas
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Online backup fails with suggested Spring transaction integration
Johan,
First of all thank you for quickly addressing my issue. I tested it and things
work just fine.
Still, I took the liberty to actually refactor your solution slightly different
hoping you could review the change and commit it to trunk. My code change
mainly frees the transaction manager from knowing anything about NeoService and
instead have the NeoService implementation (EmbeddedNeo) manage the
configuration. This also avoids casting NeoService down to EmbeddedNeo, which I
frankly have seen too many times in the code base, and really suffered under
with my own NeoService wrapper implementation ;) Anyhow, I understand that your
changes are workarounds to get things going until someone has time to properly
address the underlying transaction manager integration issue.
I hope my attached patch goes through ;)
Cheers,
-Andreas
On Wed, May 20, 2009 at 04:31:09PM +0200, Johan Svensson wrote:
Andreas,
I had a look at this and realized problem is caused by the static
nature of SpringTransactionManager and UserTransactionImpl. A fix for
this has been on my todo for some time but to make it right we need to
expose some new stuff in NeoService (mainly tied to configuration API
that has yet to be specified). For now I did a quick fix in
b9-SNAPSHOT changing the SpringTransactionManager
UserTransactionImpl classes to take NeoService in constructor.
Previously those two classes were singletons and the startup of a new
NeoService did just set a static field in them (not right). This
caused the backup slave's TM to be configured as the Spring TM and
that would in turn cause declarative transactions to fail.
I had to change your NeoServiceWrapper to take NeoService instead of a
string path to the neo db. This is because the NeoService you are
running with must be the same NeoService that was used to configure
and setup the Spring transaction manager.
I committed those changes to the test transaction project in your laboratory.
NOTE: The old suggested spring configuration (found on the wiki, use
it with neo-b8) will not work anymore when running neo-b9-SNAPSHOT.
Instead one has to do something like:
bean id=neo class=org.neo4j.api.core.EmbeddedNeo
destroy-method=shutdown lazy-init=false
constructor-arg
value${neo.store}/value
/constructor-arg
/bean
tx:annotation-driven transaction-manager=transactionManager/
alias name=neoTransactionManager alias=transactionManager/
bean id=neoTransactionManagerService
class=org.neo4j.impl.transaction.SpringTransactionManager
constructor-arg ref=neo/
/bean
bean id=neoUserTransactionService
class=org.neo4j.impl.transaction.UserTransactionImpl
constructor-arg ref=neo/
/bean
bean id=neoTransactionManager
class=org.springframework.transaction.jta.JtaTransactionManager
property name=transactionManager ref=neoTransactionManagerService/
property name=userTransaction ref=neoUserTransactionService/
/bean
Regards,
Johan
On Wed, May 20, 2009 at 2:00 AM, Andreas Guenther
aguent...@flying-orange.com wrote:
Johan, Emil,
Oki, I committed a test project resembling a simplification of what I am
using in our application.
https://svn.neo4j.org/laboratory/users/aguenther/testing/transaction
The two relevant test cases I am referring to are:
#1 ExplicitTransactionBackupTest (succeeds)
#2 SpringBasedTransactionBackupTest (fails)
The second test case relies on Spring's declarative transactions which
successfully work otherwise in our application.
I hope the code makes enough sense to spot the underlying problem. Ideally
it's just me not properly using Neo ;)
Thanks in advance,
-Andreas
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Index: src/test/org/neo4j/impl/nioneo/store/TestXa.java
===
--- src/test/org/neo4j/impl/nioneo/store/TestXa.java(revision 2869)
+++ src/test/org/neo4j/impl/nioneo/store/TestXa.java(working copy)
@@ -672,6 +672,7 @@
ds.rotateLogicalLog();
assertTrue( ds.getLogicalLog( currentVersion ) != null );
assertTrue( ds.getLogicalLog( currentVersion + 1 ) != null );
+ds.keepLogicalLogs( false );
}
catch ( Exception e )
{
Index: src/java/org/neo4j/impl/transaction/SpringTransactionManager.java
===
--- src/java/org/neo4j/impl/transaction/SpringTransactionManager.java
(revision 2869)
+++ src/java/org/neo4j/impl/transaction/SpringTransactionManager.java
(working copy)
@@ -33,12 +33,7 @@
public class SpringTransactionManager implements TransactionManager
{
-private final TransactionManager tm;
-
-public SpringTransactionManager( NeoService neo )
-{
-this.tm = ((EmbeddedNeo)
Re: [Neo] Online backup fails with suggested Spring transaction integration
On Thu, May 21, 2009 at 02:06, Andreas Guenther aguent...@flying-orange.com wrote: This also avoids casting NeoService down to EmbeddedNeo, which I frankly have seen too many times in the code base, and really suffered under with my own NeoService wrapper implementation ;) Anyhow, I understand that your changes are workarounds to get things going until someone has time to properly address the underlying transaction manager integration issue. Hi Andreas, Thanks for the patch! I'll let Johan comment on the actual code. (Though I know that he has a few fairly offline days Thu and Fri.) Just wanted to quickly jump in and say that the main -- actually hopefully the only -- reason we have all these downcasts to EmbeddedNeo is to get access to configuration stuff. And pushing out a neat config API is something I really hope we can get in to the b9 release (~mid June). So hopefully that'll sort out at least the EmbeddedNeo cast issue once and for all. Cheers, -- Emil Eifrém, CEO [e...@neotechnology.com] Neo Technology, www.neotechnology.com Cell: +46 733 462 271 | US: 206 403 8808 http://twitter.com/emileifrem ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Online backup fails with suggested Spring transaction integration
I should probably outline how spring configuration looks like with
my change. It's pretty similar to before with two constructor changes:
bean id=neoTransactionManagerService
class=org.neo4j.impl.transaction.SpringTransactionManager lazy-init=false/
bean id=neoUserTransactionService
class=org.neo4j.impl.transaction.UserTransactionImpl
constructor-arg ref=neoTransactionManagerService/
/bean
bean id=neoTransactionManager
class=org.springframework.transaction.jta.JtaTransactionManager
property name=transactionManager ref=neoTransactionManagerService/
property name=userTransaction ref=neoUserTransactionService/
/bean
bean id=neo class=org.neo4j.api.core.EmbeddedNeo
destroy-method=shutdown lazy-init=false
constructor-arg index=0 value=${neo.store}/
constructor-arg index=1 value=neoTransactionManagerService/
/bean
-Andreas
On Wed, May 20, 2009 at 05:06:56PM -0700, Andreas Guenther wrote:
Johan,
First of all thank you for quickly addressing my issue. I tested it and
things work just fine.
Still, I took the liberty to actually refactor your solution slightly
different hoping you could review the change and commit it to trunk. My code
change mainly frees the transaction manager from knowing anything about
NeoService and instead have the NeoService implementation (EmbeddedNeo)
manage the configuration. This also avoids casting NeoService down to
EmbeddedNeo, which I frankly have seen too many times in the code base, and
really suffered under with my own NeoService wrapper implementation ;)
Anyhow, I understand that your changes are workarounds to get things going
until someone has time to properly address the underlying transaction manager
integration issue.
I hope my attached patch goes through ;)
Cheers,
-Andreas
On Wed, May 20, 2009 at 04:31:09PM +0200, Johan Svensson wrote:
Andreas,
I had a look at this and realized problem is caused by the static
nature of SpringTransactionManager and UserTransactionImpl. A fix for
this has been on my todo for some time but to make it right we need to
expose some new stuff in NeoService (mainly tied to configuration API
that has yet to be specified). For now I did a quick fix in
b9-SNAPSHOT changing the SpringTransactionManager
UserTransactionImpl classes to take NeoService in constructor.
Previously those two classes were singletons and the startup of a new
NeoService did just set a static field in them (not right). This
caused the backup slave's TM to be configured as the Spring TM and
that would in turn cause declarative transactions to fail.
I had to change your NeoServiceWrapper to take NeoService instead of a
string path to the neo db. This is because the NeoService you are
running with must be the same NeoService that was used to configure
and setup the Spring transaction manager.
I committed those changes to the test transaction project in your
laboratory.
NOTE: The old suggested spring configuration (found on the wiki, use
it with neo-b8) will not work anymore when running neo-b9-SNAPSHOT.
Instead one has to do something like:
bean id=neo class=org.neo4j.api.core.EmbeddedNeo
destroy-method=shutdown lazy-init=false
constructor-arg
value${neo.store}/value
/constructor-arg
/bean
tx:annotation-driven transaction-manager=transactionManager/
alias name=neoTransactionManager alias=transactionManager/
bean id=neoTransactionManagerService
class=org.neo4j.impl.transaction.SpringTransactionManager
constructor-arg ref=neo/
/bean
bean id=neoUserTransactionService
class=org.neo4j.impl.transaction.UserTransactionImpl
constructor-arg ref=neo/
/bean
bean id=neoTransactionManager
class=org.springframework.transaction.jta.JtaTransactionManager
property name=transactionManager ref=neoTransactionManagerService/
property name=userTransaction ref=neoUserTransactionService/
/bean
Regards,
Johan
On Wed, May 20, 2009 at 2:00 AM, Andreas Guenther
aguent...@flying-orange.com wrote:
Johan, Emil,
Oki, I committed a test project resembling a simplification of what I am
using in our application.
https://svn.neo4j.org/laboratory/users/aguenther/testing/transaction
The two relevant test cases I am referring to are:
#1 ExplicitTransactionBackupTest (succeeds)
#2 SpringBasedTransactionBackupTest (fails)
The second test case relies on Spring's declarative transactions which
successfully work otherwise in our application.
I hope the code makes enough sense to spot the underlying problem.
Ideally it's just me not properly using Neo ;)
Thanks in advance,
-Andreas
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Index:
Re: [Neo] Online backup fails with suggested Spring transaction integration
Emil, I'd prefer to upload what I have to the laboratory once my CLA is accepted and a sub folder is available. This will make it easier to keep all my posted issues and testing code in the Neo space for future reference. -Andreas Emil Eifrem wrote: On Sun, May 17, 2009 at 23:45, Andreas Guenther aguent...@flying-orange.com wrote: Johan, Looks like there's no unexpected shutdown going on according to my debugging. I do have a full test project available for you but it's too big for pasting it here. Any suggestion on where I can place you a Maven test project tarball? Hi Andreas, Well, I'd suggest using something like dropbox [1] or yousendit [2]. Or if you do want to submit patches as discussed previously and agree with our contributor policy [3], then you can just sign the CLA [4]. After that, I'd be glad to give you write access to the svn repo and you can add it to your laboratory. 1] http://getdropbox.com 2] http://yousendit.com/ 3] http://wiki.neo4j.org/content/Code_Contributor%27s_Guide 4] http://wiki.neo4j.org/content/About_Contributor_License_Agreement Cheers, ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Online backup fails with suggested Spring transaction integration
Johan, Emil, Oki, I committed a test project resembling a simplification of what I am using in our application. https://svn.neo4j.org/laboratory/users/aguenther/testing/transaction The two relevant test cases I am referring to are: #1 ExplicitTransactionBackupTest (succeeds) #2 SpringBasedTransactionBackupTest (fails) The second test case relies on Spring's declarative transactions which successfully work otherwise in our application. I hope the code makes enough sense to spot the underlying problem. Ideally it's just me not properly using Neo ;) Thanks in advance, -Andreas On Tue, May 19, 2009 at 11:34:21PM +0200, Emil Eifrem wrote: On Tue, May 19, 2009 at 18:17, Andreas Guenther aguent...@flying-orange.com wrote: Emil, I'd prefer to upload what I have to the laboratory once my CLA is accepted and a sub folder is available. This will make it easier to keep all my posted issues and testing code in the Neo space for future reference. Hi Andreas, Good that you brought this up, cause the CLA you sent was actually caught by Google's spam filters! :( I'll set it up for you immediately. Cheers, -- Emil Eifrém, CEO [e...@neotechnology.com] Neo Technology, www.neotechnology.com Cell: +46 733 462 271 | US: 206 403 8808 http://twitter.com/emileifrem ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Online backup fails with suggested Spring transaction integration
On Sat, May 16, 2009 at 10:25 PM, Andreas Guenther aguent...@flying-orange.com wrote: I have been poking around with online backup for a couple of days now and are happy with the results in general, especially the runtime backup capability. Most of the issues I came across I was able to work around but there is one obstacles left that I need some help on. As long as I use Neo's transactions programmatically taking backups work just fine at any given point in time during regular Neo usage. But once I switch to declarative transactions through the suggested Spring configuration taken from the IMdb example I am running into all sorts of exceptions after the first backup is taken. Below is the Spring configuration and some exceptions I got. If helpful I could extend the online-backup test suite with declarative transactions. Could there be some log file access conflicts going on? It's probably worth mentioning that I am using declarative transactions throughout the rest of our app and it's been working just great after many stress tests and without using online backup. Any thoughts on this one? -Andreas Looks like the global transaction log has been closed. Can you verify that NeoService.shutdown() hasn't been invoked? Another possible cause could be wrong use of Thread.interrupt() somewhere. Would be great if you could produce a test case for this problem so I can have a look at it. Regards, Johan ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Online backup fails with suggested Spring transaction integration
Johan, Looks like there's no unexpected shutdown going on according to my debugging. I do have a full test project available for you but it's too big for pasting it here. Any suggestion on where I can place you a Maven test project tarball? -Andreas Johan Svensson wrote: On Sat, May 16, 2009 at 10:25 PM, Andreas Guenther aguent...@flying-orange.com wrote: I have been poking around with online backup for a couple of days now and are happy with the results in general, especially the runtime backup capability. Most of the issues I came across I was able to work around but there is one obstacles left that I need some help on. As long as I use Neo's transactions programmatically taking backups work just fine at any given point in time during regular Neo usage. But once I switch to declarative transactions through the suggested Spring configuration taken from the IMdb example I am running into all sorts of exceptions after the first backup is taken. Below is the Spring configuration and some exceptions I got. If helpful I could extend the online-backup test suite with declarative transactions. Could there be some log file access conflicts going on? It's probably worth mentioning that I am using declarative transactions throughout the rest of our app and it's been working just great after many stress tests and without using online backup. Any thoughts on this one? -Andreas Looks like the global transaction log has been closed. Can you verify that NeoService.shutdown() hasn't been invoked? Another possible cause could be wrong use of Thread.interrupt() somewhere. Would be great if you could produce a test case for this problem so I can have a look at it. Regards, Johan ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Online backup fails with suggested Spring transaction integration
On Sun, May 17, 2009 at 23:45, Andreas Guenther aguent...@flying-orange.com wrote: Johan, Looks like there's no unexpected shutdown going on according to my debugging. I do have a full test project available for you but it's too big for pasting it here. Any suggestion on where I can place you a Maven test project tarball? Hi Andreas, Well, I'd suggest using something like dropbox [1] or yousendit [2]. Or if you do want to submit patches as discussed previously and agree with our contributor policy [3], then you can just sign the CLA [4]. After that, I'd be glad to give you write access to the svn repo and you can add it to your laboratory. 1] http://getdropbox.com 2] http://yousendit.com/ 3] http://wiki.neo4j.org/content/Code_Contributor%27s_Guide 4] http://wiki.neo4j.org/content/About_Contributor_License_Agreement Cheers, -- Emil Eifrém, CEO [e...@neotechnology.com] Neo Technology, www.neotechnology.com Cell: +46 733 462 271 | US: 206 403 8808 http://twitter.com/emileifrem ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] Online backup fails with suggested Spring transaction integration
I have been poking around with online backup for a couple of days now and are happy with the results in general, especially the runtime backup capability. Most of the issues I came across I was able to work around but there is one obstacles left that I need some help on. As long as I use Neo's transactions programmatically taking backups work just fine at any given point in time during regular Neo usage. But once I switch to declarative transactions through the suggested Spring configuration taken from the IMdb example I am running into all sorts of exceptions after the first backup is taken. Below is the Spring configuration and some exceptions I got. If helpful I could extend the online-backup test suite with declarative transactions. Could there be some log file access conflicts going on? It's probably worth mentioning that I am using declarative transactions throughout the rest of our app and it's been working just great after many stress tests and without using online backup. Any thoughts on this one? -Andreas --- Spring configuration --- bean id=neoTransactionManagerService class=org.neo4j.impl.transaction.SpringTransactionManager/ bean id=neoUserTransactionService class=org.neo4j.impl.transaction.UserTransactionImpl/ bean id=neoTransactionManager class=org.springframework.transaction.jta.JtaTransactionManager property name=transactionManager ref=neoTransactionManagerService/ property name=userTransaction ref=neoUserTransactionService/ /bean --- --- Exceptions --- java.nio.channels.ClosedChannelException at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91) at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:192) at org.neo4j.impl.transaction.TxLog.txStart(TxLog.java:143) at org.neo4j.impl.transaction.TxManager.begin(TxManager.java:502) at org.neo4j.impl.transaction.UserTransactionImpl.begin(UserTransactionImpl.java:47) at org.springframework.transaction.jta.JtaTransactionManager.doJtaBegin(JtaTransactionManager.java:886) at org.springframework.transaction.jta.JtaTransactionManager.doBegin(JtaTransactionManager.java:843) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:374) ... org.springframework.transaction.CannotCreateTransactionException: JTA failure on begin; nested exception is javax.transaction.SystemException: TM encountered a problem, error writing transaction log,java.nio.channels.ClosedChannelException at org.springframework.transaction.jta.JtaTransactionManager.doBegin(JtaTransactionManager.java:856) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:374) ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
