[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417601#comment-16417601
 ] 

ASF subversion and git services commented on ARTEMIS-1541:
--

Commit f005da6dfdf4d51b244e360abdf1906f7d45670d in activemq-artemis's branch 
refs/heads/1.x from [~nigro@gmail.com]
[ https://git-wip-us.apache.org/repos/asf?p=activemq-artemis.git;h=f005da6 ]

ARTEMIS-1541 Make the JDBC Node Manager more resilient on failures

(cherry picked from commit 70b21725edae28b591b87bb4de0f51364e9cfd50)


> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>Priority: Major
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299093#comment-16299093
 ] 

ASF GitHub Bot commented on ARTEMIS-1541:
-

Github user asfgit closed the pull request at:

https://github.com/apache/activemq-artemis/pull/1691


> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299091#comment-16299091
 ] 

ASF subversion and git services commented on ARTEMIS-1541:
--

Commit 170e89f9fccd7ec7b3d10440dd2579921d7bf2c5 in activemq-artemis's branch 
refs/heads/master from [~nigro@gmail.com]
[ https://git-wip-us.apache.org/repos/asf?p=activemq-artemis.git;h=170e89f ]

ARTEMIS-1541 Make the JDBC Node Manager more resilient on failures

In order to make the JDBC Node Manager more resilient has been implemented:
- recovering with fixed number of retries during the NodeId setup + 
unrecoverable failure otherwise
- unrecoverable fail on exceptions while renewing a lease lock

In addition, in different parts of these critical processes are added more log 
informations to help diagnose.


> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292291#comment-16292291
 ] 

ASF GitHub Bot commented on ARTEMIS-1541:
-

Github user franz1981 commented on the issue:

https://github.com/apache/activemq-artemis/pull/1691
  
@mtaylor 
The DBMS I've tested are MySQL 5.7 and Oracle 12 c (and obviously Derby!).
What I think it worths to do is improve the tests to make them more 
configurations with ease but I'm thinking to address it as a separate PR in 
order to not charge this one with other changes, wdyt?



> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292287#comment-16292287
 ] 

ASF GitHub Bot commented on ARTEMIS-1541:
-

Github user mtaylor commented on the issue:

https://github.com/apache/activemq-artemis/pull/1691
  
> Do you mean which versions of which DBMS?
Yes.

>  how do you propose to do that?
Just add the driver to the class path and pass in the driver class 
parameter via system properties.  We do this for other JDBC tests


> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292237#comment-16292237
 ] 

ASF GitHub Bot commented on ARTEMIS-1541:
-

Github user franz1981 commented on the issue:

https://github.com/apache/activemq-artemis/pull/1691
  
@mtaylor 
> and the thread does not cause interference with other tests

That's the part I'm sure of it: it seems that only a SIGKILL can close a 
process that has any of these threads running-

> Which systems did you run the tests against? 

Do you mean which versions of which DBMS?

> It would be good to have this set up in the CI env

Considering that I'm not sure that is possible to kill gently the Oracle 
one, how do you propose to do that?
eg The Oracle thin driver is not on maven AFAIK




> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292230#comment-16292230
 ] 

ASF GitHub Bot commented on ARTEMIS-1541:
-

Github user mtaylor commented on the issue:

https://github.com/apache/activemq-artemis/pull/1691
  
@franz1981 If you really can't kill it and the thread does not cause 
interference with other tests you can add an exception to the 
ThreadCheckerRule.  Which systems did you run the tests against?  It would be 
good to have this set up in the CI env.


> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292168#comment-16292168
 ] 

ASF GitHub Bot commented on ARTEMIS-1541:
-

Github user franz1981 commented on the issue:

https://github.com/apache/activemq-artemis/pull/1691
  
@mtaylor I've performed the failover tests on other DBMS and it is working 
as expected, but disabling the `ThreadLeakCheckRule` due to a "known" leak on 
the Oracle thin driver - the uninterruptible/unstoppable/unbreakable 
`oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser` 
thread:
```
public void run() {
while(true) {
try {

BlockSource.ThreadedCachingBlockSource.BlockReleaser var1 = SOLE_INSTANCE;
synchronized(SOLE_INSTANCE) {
SOLE_INSTANCE.wait(30L);
}


BlockSource.ThreadedCachingBlockSource.releaseFromAllSources();
} catch (Throwable var4) {
;
}
}
}
```



> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292159#comment-16292159
 ] 

ASF GitHub Bot commented on ARTEMIS-1541:
-

GitHub user franz1981 reopened a pull request:

https://github.com/apache/activemq-artemis/pull/1691

ARTEMIS-1541 Make the JDBC Node Manager more resilient on failures

In order to make the JDBC Node Manager more resilient has been implemented:
- recovering with fixed number of retries during the NodeId setup + 
unrecoverable failure otherwise
- unrecoverable fail on exceptions while renewing a lease lock

In addition, in different parts of these critical processes are added more 
log informations to help diagnose.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/franz1981/activemq-artemis 
jdbc_ha_improvements

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/1691.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1691


commit 70b21725edae28b591b87bb4de0f51364e9cfd50
Author: Francesco Nigro 
Date:   2017-12-06T10:28:59Z

ARTEMIS-1541 Make the JDBC Node Manager more resilient on failures

In order to make the JDBC Node Manager more resilient has been implemented:
- recovering with fixed number of retries during the NodeId setup + 
unrecoverable failure otherwise
- unrecoverable fail on exceptions while renewing a lease lock

In addition, in different parts of these critical processes are added more 
log informations to help diagnose.




> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281814#comment-16281814
 ] 

ASF GitHub Bot commented on ARTEMIS-1541:
-

Github user franz1981 closed the pull request at:

https://github.com/apache/activemq-artemis/pull/1691


> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281813#comment-16281813
 ] 

ASF GitHub Bot commented on ARTEMIS-1541:
-

Github user franz1981 commented on the issue:

https://github.com/apache/activemq-artemis/pull/1691
  
@mtaylor Thanks! And good point: I need to do further tests with the 
different DBMS first :+1: 


> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281810#comment-16281810
 ] 

ASF GitHub Bot commented on ARTEMIS-1541:
-

Github user mtaylor commented on the issue:

https://github.com/apache/activemq-artemis/pull/1691
  
@franz1981 nice one.  Did you test this against all our current supported 
databases?  Our CI currently doesn't cover DB tests.


> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280885#comment-16280885
 ] 

ASF GitHub Bot commented on ARTEMIS-1541:
-

Github user franz1981 commented on the issue:

https://github.com/apache/activemq-artemis/pull/1691
  
@mtaylor The CI tests hasn't shown any regression due to this PR :+1: 


> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280456#comment-16280456
 ] 

ASF GitHub Bot commented on ARTEMIS-1541:
-

Github user franz1981 commented on the issue:

https://github.com/apache/activemq-artemis/pull/1691
  
@mtaylor Please do not merge it yet: I need to run the CI tests first 

@mtaylor @jmesnil How it seems? 



> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ARTEMIS-1541) Make the JDBC Node Manager more resilient on failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ARTEMIS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280455#comment-16280455
 ] 

ASF GitHub Bot commented on ARTEMIS-1541:
-

GitHub user franz1981 opened a pull request:

https://github.com/apache/activemq-artemis/pull/1691

ARTEMIS-1541 Make the JDBC Node Manager more resilient on failures

In order to make the JDBC Node Manager more resilient has been implemented:
- lowered from SERIALIZABLE to READ_COMMITTED Isolation level on NodeId 
setup to avoid
  ad-hoc (ie per DBMS) handling of failures
- recovering with fixed number of retries during the NodeId setup + 
unrecoverable failure otherwise
- unrecoverable fail on exceptions while renewing a lease lock

In addition, in different parts of these critical processes are added more 
log informations to help diagnose.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/franz1981/activemq-artemis 
jdbc_ha_improvements

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/1691.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1691






> Make the JDBC Node Manager more resilient on failures
> -
>
> Key: ARTEMIS-1541
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1541
> Project: ActiveMQ Artemis
>  Issue Type: Improvement
>  Components: Broker
>Reporter: Francesco Nigro
>Assignee: Francesco Nigro
>
> The JDBC Node Manager isn't showing (at any level) errors on setup and hasn't 
> any policy to fail or recover while executing certain critical operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)