[
https://issues.apache.org/jira/browse/IMPALA-12470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17778000#comment-17778000
]
Wenzhe Zhou edited comment on IMPALA-12470 at 10/21/23 3:28 AM:
----------------------------------------------------------------
test_ext_data_sourcespy::TestExtDataSources::test_jdbc_data_source failed with
exception on ozone with following stack:
{code:java}
E1011 06:08:01.009253 11969 GenericJdbcDatabaseAccessor.java:145]
f248473be4b7e2d2:4c92512800000001] Caught exception while trying to execute
query
Java exception follows:
java.util.concurrent.ExecutionException:
org.apache.impala.extdatasource.jdbc.exception.JdbcDatabaseAccessException:
Unable to cache jdbc driver jar at location
'/test-warehouse/data-sources/jdbc-drivers/postgresql-jdbc.jar'. Check that the
file exists and is readable. Message: [Failed to get file info
/test-warehouse/data-sources/jdbc-drivers/postgresql-jdbc.jar
Error(2): No such file or directory]
at
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588)
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547)
at
com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:113)
at
com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:240)
at
com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2317)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2283)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
at com.google.common.cache.LocalCache.get(LocalCache.java:3966)
at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4863)
at
org.apache.impala.extdatasource.jdbc.dao.GenericJdbcDatabaseAccessor.initializeDatabaseSource(GenericJdbcDatabaseAccessor.java:226)
at
org.apache.impala.extdatasource.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:131)
at
org.apache.impala.extdatasource.jdbc.JdbcDataSource.buildQueryAndExecute(JdbcDataSource.java:306)
at
org.apache.impala.extdatasource.jdbc.JdbcDataSource.open(JdbcDataSource.java:158)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.open(ExternalDataSourceExecutor.java:269)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.open(ExternalDataSourceExecutor.java:212)
Caused by:
org.apache.impala.extdatasource.jdbc.exception.JdbcDatabaseAccessException:
Unable to cache jdbc driver jar at location
'/test-warehouse/data-sources/jdbc-drivers/postgresql-jdbc.jar'. Check that the
file exists and is readable. Message: [Failed to get file info
/test-warehouse/data-sources/jdbc-drivers/postgresql-jdbc.jar
Error(2): No such file or directory]
at
org.apache.impala.extdatasource.jdbc.dao.GenericJdbcDatabaseAccessor.lambda$initializeDatabaseSource$1(GenericJdbcDatabaseAccessor.java:237)
at
com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4868)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282)
... 10 more
E1011 06:08:01.009610 11969 ExternalDataSourceExecutor.java:253]
f248473be4b7e2d2:4c92512800000001] Error in data source
(path=/tmp/jdbc-data-source.3552.13.jar,
class=org.apache.impala.extdatasource.jdbc.JdbcDataSource, version=V1) close():
No error message returned by data source. Check the impalad log for more
information.
Java exception follows:
java.lang.IllegalArgumentException
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:131)
at
org.apache.impala.extdatasource.jdbc.JdbcDataSource.close(JdbcDataSource.java:211)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.close(ExternalDataSourceExecutor.java:289)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.close(ExternalDataSourceExecutor.java:234)
{code}
In GenericJdbcDatabaseAccessor.java:226, dbcpDataSource =
dataSourceCache.get(cacheMapKey) failed
For test_ext_data_sources.TestExtDataSources.test_verify_jdbc_table_properties,
expected jdbc jar url on Ozone is:
test_ext_data_sources.TestExtDataSources.test_verify_jdbc_table_properties
{code:java}
query_test/test_ext_data_sources.py:70: in test_verify_jdbc_table_properties
assert properties['__IMPALA_DATA_SOURCE_LOCATION'] == \
E assert 'ofs://localh...ta-source.jar' == 'hdfs://localh...ta-source.jar'
E -
ofs://localhost:9862/impala/test-warehouse/data-sources/jdbc-data-source.jar
E ? ^ --- ^^^^^^^
E + hdfs://localhost:20500/test-warehouse/data-sources/jdbc-data-source.jar
E ? ^^ ^^^^
{code}
was (Author: wzhou):
test_ext_data_sourcespy::TestExtDataSources::test_jdbc_data_source failed with
exception on ozone with following stack:
{code:java}
E1011 06:08:01.009253 11969 GenericJdbcDatabaseAccessor.java:145]
f248473be4b7e2d2:4c92512800000001] Caught exception while trying to execute
query
Java exception follows:
java.util.concurrent.ExecutionException:
org.apache.impala.extdatasource.jdbc.exception.JdbcDatabaseAccessException:
Unable to cache jdbc driver jar at location
'/test-warehouse/data-sources/jdbc-drivers/postgresql-jdbc.jar'. Check that the
file exists and is readable. Message: [Failed to get file info
/test-warehouse/data-sources/jdbc-drivers/postgresql-jdbc.jar
Error(2): No such file or directory]
at
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588)
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547)
at
com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:113)
at
com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:240)
at
com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2317)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2283)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
at com.google.common.cache.LocalCache.get(LocalCache.java:3966)
at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4863)
at
org.apache.impala.extdatasource.jdbc.dao.GenericJdbcDatabaseAccessor.initializeDatabaseSource(GenericJdbcDatabaseAccessor.java:226)
at
org.apache.impala.extdatasource.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:131)
at
org.apache.impala.extdatasource.jdbc.JdbcDataSource.buildQueryAndExecute(JdbcDataSource.java:306)
at
org.apache.impala.extdatasource.jdbc.JdbcDataSource.open(JdbcDataSource.java:158)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.open(ExternalDataSourceExecutor.java:269)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.open(ExternalDataSourceExecutor.java:212)
Caused by:
org.apache.impala.extdatasource.jdbc.exception.JdbcDatabaseAccessException:
Unable to cache jdbc driver jar at location
'/test-warehouse/data-sources/jdbc-drivers/postgresql-jdbc.jar'. Check that the
file exists and is readable. Message: [Failed to get file info
/test-warehouse/data-sources/jdbc-drivers/postgresql-jdbc.jar
Error(2): No such file or directory]
at
org.apache.impala.extdatasource.jdbc.dao.GenericJdbcDatabaseAccessor.lambda$initializeDatabaseSource$1(GenericJdbcDatabaseAccessor.java:237)
at
com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4868)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282)
... 10 more
E1011 06:08:01.009610 11969 ExternalDataSourceExecutor.java:253]
f248473be4b7e2d2:4c92512800000001] Error in data source
(path=/tmp/jdbc-data-source.3552.13.jar,
class=org.apache.impala.extdatasource.jdbc.JdbcDataSource, version=V1) close():
No error message returned by data source. Check the impalad log for more
information.
Java exception follows:
java.lang.IllegalArgumentException
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:131)
at
org.apache.impala.extdatasource.jdbc.JdbcDataSource.close(JdbcDataSource.java:211)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.close(ExternalDataSourceExecutor.java:289)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.close(ExternalDataSourceExecutor.java:234)
{code}
In GenericJdbcDatabaseAccessor.java:226, dbcpDataSource =
dataSourceCache.get(cacheMapKey) failed
For test_ext_data_sources.TestExtDataSources.test_verify_jdbc_table_properties,
expect jdbc jar url is:
test_ext_data_sources.TestExtDataSources.test_verify_jdbc_table_properties
{code:java}
query_test/test_ext_data_sources.py:70: in test_verify_jdbc_table_properties
assert properties['__IMPALA_DATA_SOURCE_LOCATION'] == \
E assert 'ofs://localh...ta-source.jar' == 'hdfs://localh...ta-source.jar'
E -
ofs://localhost:9862/impala/test-warehouse/data-sources/jdbc-data-source.jar
E ? ^ --- ^^^^^^^
E + hdfs://localhost:20500/test-warehouse/data-sources/jdbc-data-source.jar
E ? ^^ ^^^^
{code}
> Support different schemes for jdbc driver url when creating external jdbc
> table
> -------------------------------------------------------------------------------
>
> Key: IMPALA-12470
> URL: https://issues.apache.org/jira/browse/IMPALA-12470
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
> Reporter: Wenzhe Zhou
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> jdbc driver url is specified as table property 'driver.url' when creating a
> external jdbc table, like the sample below:
> {code:java}
> CREATE TABLE test_postgres (
> id INT,
> string_col STRING)
> PRODUCED BY DATA SOURCE JdbcDataSource(
> '{"database.type":"POSTGRES",
> "jdbc.url":"jdbc:postgresql://localhost:5432/functional",
> "jdbc.driver":"org.postgresql.Driver",
> "driver.url":"hdfs://localhost:20500/test-warehouse/data-sources/jdbc-driver/postgresql-jdbc.jar",
> "dbcp.username":"hiveuser",
> "dbcp.password":"password",
> "table":"alltypes"}');
> {code}
> In the initial patch, we only support to download the driver jar files from
> hdfs. We need to support different storage type, like 'file://', 's3a://',
> etc.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
