ClownfishYang opened a new issue #10618:
URL: https://github.com/apache/druid/issues/10618
My application is using multiple DataSource, where the OrientDB is the
non-primary data source and hopefully will not affect the main application when
it is not available, but unfortunately it does not.
### Affected Version
1.2.3
### Description
### Error Log
```
2020-11-30 18:01:48 [Druid-ConnectionPool-Create-1288163341] ERROR
com.alibaba.druid.pool.DruidDataSource - create connection RuntimeException
com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open
database 'orient_db_pro'
at
com.orientechnologies.orient.core.db.OrientDBRemote.open(OrientDBRemote.java:89)
at
com.orientechnologies.orient.core.db.OrientDB.open(OrientDB.java:205)
at
com.orientechnologies.orient.core.db.OrientDB.open(OrientDB.java:190)
at
com.orientechnologies.orient.jdbc.OrientJdbcConnection.<init>(OrientJdbcConnection.java:68)
at
com.orientechnologies.orient.jdbc.OrientJdbcDriver.connect(OrientJdbcDriver.java:52)
at
com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:156)
at
com.alibaba.druid.filter.FilterAdapter.connection_connect(FilterAdapter.java:787)
at
com.alibaba.druid.filter.FilterEventAdapter.connection_connect(FilterEventAdapter.java:38)
at
com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:150)
at
com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1646)
at
com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1710)
at
com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2774)
Caused by: com.orientechnologies.orient.core.exception.OStorageException:
Cannot create a connection to remote server address(es): [10.xx.xx.xx:2424]
DB name="orient_db_pro"
at
com.orientechnologies.orient.client.remote.OStorageRemote.openRemoteDatabase(OStorageRemote.java:1700)
at
com.orientechnologies.orient.client.remote.OStorageRemote.openRemoteDatabase(OStorageRemote.java:1554)
at
com.orientechnologies.orient.client.remote.OStorageRemote.open(OStorageRemote.java:521)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentRemote.internalOpen(ODatabaseDocumentRemote.java:225)
at
com.orientechnologies.orient.core.db.OrientDBRemote.open(OrientDBRemote.java:86)
... 11 common frames omitted
2020-11-30 18:01:48 [Druid-ConnectionPool-Create-1288163341] INFO
com.alibaba.druid.pool.DruidAbstractDataSource - {dataSource-6} failContinuous
is true
2020-11-30 18:01:48 [Druid-ConnectionPool-Create-1288163341] ERROR
com.alibaba.druid.pool.DruidDataSource - create connection RuntimeException
com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open
database 'orient_db_pro'
at
com.orientechnologies.orient.core.db.OrientDBRemote.open(OrientDBRemote.java:89)
at
com.orientechnologies.orient.core.db.OrientDB.open(OrientDB.java:205)
at
com.orientechnologies.orient.core.db.OrientDB.open(OrientDB.java:190)
at
com.orientechnologies.orient.jdbc.OrientJdbcConnection.<init>(OrientJdbcConnection.java:68)
at
com.orientechnologies.orient.jdbc.OrientJdbcDriver.connect(OrientJdbcDriver.java:52)
at
com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:156)
at
com.alibaba.druid.filter.FilterAdapter.connection_connect(FilterAdapter.java:787)
at
com.alibaba.druid.filter.FilterEventAdapter.connection_connect(FilterEventAdapter.java:38)
at
com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:150)
at
com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1646)
at
com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1710)
at
com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2774)
Caused by: com.orientechnologies.orient.core.exception.OStorageException:
Cannot create a connection to remote server address(es): [10.xx.xx.xx:2424]
DB name="orient_db_pro"
at
com.orientechnologies.orient.client.remote.OStorageRemote.openRemoteDatabase(OStorageRemote.java:1700)
at
com.orientechnologies.orient.client.remote.OStorageRemote.openRemoteDatabase(OStorageRemote.java:1554)
at
com.orientechnologies.orient.client.remote.OStorageRemote.open(OStorageRemote.java:521)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentRemote.internalOpen(ODatabaseDocumentRemote.java:225)
at
com.orientechnologies.orient.core.db.OrientDBRemote.open(OrientDBRemote.java:86)
... 11 common frames omitted
2020-11-30 18:01:48 [Druid-ConnectionPool-Create-1288163341] ERROR
com.alibaba.druid.pool.DruidDataSource - create connection RuntimeException
com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open
database 'orient_db_pro'
at
com.orientechnologies.orient.core.db.OrientDBRemote.open(OrientDBRemote.java:89)
at
com.orientechnologies.orient.core.db.OrientDB.open(OrientDB.java:205)
at
com.orientechnologies.orient.core.db.OrientDB.open(OrientDB.java:190)
at
com.orientechnologies.orient.jdbc.OrientJdbcConnection.<init>(OrientJdbcConnection.java:68)
at
com.orientechnologies.orient.jdbc.OrientJdbcDriver.connect(OrientJdbcDriver.java:52)
at
com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:156)
at
com.alibaba.druid.filter.FilterAdapter.connection_connect(FilterAdapter.java:787)
at
com.alibaba.druid.filter.FilterEventAdapter.connection_connect(FilterEventAdapter.java:38)
at
com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:150)
at
com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1646)
at
com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1710)
at
com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2774)
Caused by: com.orientechnologies.orient.core.exception.OStorageException:
Cannot create a connection to remote server address(es): [10.xx.xx.xx:2424]
DB name="orient_db_pro"
at
com.orientechnologies.orient.client.remote.OStorageRemote.openRemoteDatabase(OStorageRemote.java:1700)
at
com.orientechnologies.orient.client.remote.OStorageRemote.openRemoteDatabase(OStorageRemote.java:1554)
at
com.orientechnologies.orient.client.remote.OStorageRemote.open(OStorageRemote.java:521)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentRemote.internalOpen(ODatabaseDocumentRemote.java:225)
at
com.orientechnologies.orient.core.db.OrientDBRemote.open(OrientDBRemote.java:86)
... 11 common frames omitted
```
### Problem analysis
From the exception stack, you can see that CreateConnectionTask is always
running to create a connection, but always fails.
Since the OrientDB driver was throwing a RuntimeException when creating the
connection was unavailable, the OrientDB was constantly trying again until the
program was unavailable.
The problem code:
```
private void runInternal() {
for (;;) {
// addLast
...
PhysicalConnectionInfo physicalConnection = null;
try {
physicalConnection = createPhysicalConnection();
} catch (OutOfMemoryError e) {
LOG.error("create connection OutOfMemoryError, out
memory. ", e);
...
} catch (SQLException e) {
LOG.error("create connection SQLException, url: " +
jdbcUrl, e);
errorCount++;
if (errorCount > connectionErrorRetryAttempts &&
timeBetweenConnectErrorMillis > 0) {
// fail over retry attempts
setFailContinuous(true);
if (failFast) {
lock.lock();
try {
notEmpty.signalAll();
} finally {
lock.unlock();
}
}
if (breakAfterAcquireFailure) {
lock.lock();
try {
clearCreateTask(taskId);
} finally {
lock.unlock();
}
return;
}
this.errorCount = 0; // reset errorCount
if (closing || closed) {
lock.lock();
try {
clearCreateTask(taskId);
} finally {
lock.unlock();
}
return;
}
createSchedulerFuture =
createScheduler.schedule(this, timeBetweenConnectErrorMillis,
TimeUnit.MILLISECONDS);
return;
}
} catch (RuntimeException e) {
LOG.error("create connection RuntimeException", e);
// unknow fatal exception
setFailContinuous(true);
continue;
} catch (Error e) {
lock.lock();
try {
clearCreateTask(taskId);
} finally {
lock.unlock();
}
LOG.error("create connection Error", e);
// unknow fatal exception
setFailContinuous(true);
break;
} catch (Throwable e) {
lock.lock();
try {
clearCreateTask(taskId);
} finally {
lock.unlock();
}
LOG.error("create connection unexecpted error.", e);
break;
}
...
}
}
```
I have a few questions about this:
1. Why `Druid` use continue instead of break when handling
`RuntimeException`;
2. Whether there is a need for the parameter of `breakAfterAcquireFailure`,
maybe we need a parameter of the maximum number of retry;
3. This is about the `OrientDB`. Why would the driver throw a
`RuntimeException` instead of `SQLException`;
I will Issue these questions on git of both parties. If changes are needed,
I think I can achieve them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
