[
https://issues.apache.org/jira/browse/IMPALA-12470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17778004#comment-17778004
]
Wenzhe Zhou edited comment on IMPALA-12470 at 10/21/23 5:35 PM:
----------------------------------------------------------------
For
query_test.test_ext_data_sources.TestExtDataSources.test_verify_jdbc_table_properties,
expected jdbc jar url on S3 is:
s3a://impala-test-uswest2-2/test-warehouse/data-sources/jdbc-data-source.jar
{code:java}
query_test/test_ext_data_sources.py:70: in test_verify_jdbc_table_properties
assert properties['__IMPALA_DATA_SOURCE_LOCATION'] == \
E assert 's3a://impala...ta-source.jar' == 'hdfs://localh...ta-source.jar'
E -
s3a://impala-test-uswest2-2/test-warehouse/data-sources/jdbc-data-source.jar
E ? -- ^^^ ^^^^ ^^^^^^^ ^^
E + hdfs://localhost:20500/test-warehouse/data-sources/jdbc-data-source.jar
E ? +++ ^^^ ^^ ^ ^^^^
{code}
Stack trace of the test failure of
test_ext_data_sources.TestExtDataSources.test_jdbc_data_source
{code:java}
1012 05:26:05.270896 1181 GenericJdbcDatabaseAccessor.java:145]
c64d3ba0d79d1aec:076d0d8f00000001] Caught exception while trying to execute
query
Java exception follows:
java.sql.SQLException: Cannot create PoolableConnectionFactory (FATAL: database
"functional" does not exist)
at
org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:669)
at
org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:544)
at
org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:753)
at
org.apache.impala.extdatasource.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:137)
at
org.apache.impala.extdatasource.jdbc.JdbcDataSource.buildQueryAndExecute(JdbcDataSource.java:306)
at
org.apache.impala.extdatasource.jdbc.JdbcDataSource.open(JdbcDataSource.java:158)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.open(ExternalDataSourceExecutor.java:269)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.open(ExternalDataSourceExecutor.java:212)
Caused by: org.postgresql.util.PSQLException: FATAL: database "functional" does
not exist
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676)
at
org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2788)
at
org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:174)
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:290)
at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:247)
at org.postgresql.Driver.makeConnection(Driver.java:434)
at org.postgresql.Driver.connect(Driver.java:291)
at
org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:55)
at
org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:355)
at
org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:115)
at
org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:665)
... 7 more
E1012 05:26:05.271389 1181 ExternalDataSourceExecutor.java:253]
c64d3ba0d79d1aec:076d0d8f00000001] Error in data source
(path=/tmp/jdbc-data-source.6397.13.jar,
class=org.apache.impala.extdatasource.jdbc.JdbcDataSource, version=V1) close():
No error message returned by data source. Check the impalad log for more
information.
Java exception follows:
java.lang.IllegalArgumentException
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:131)
at
org.apache.impala.extdatasource.jdbc.JdbcDataSource.close(JdbcDataSource.java:211)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.close(ExternalDataSourceExecutor.java:289)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.close(ExternalDataSourceExecutor.java:234)
{code}
It seems functional database is not created on Postgres server. That means
testdata/bin/load-ext-data-sources.sh was not ran successfully on s3.
was (Author: wzhou):
For
query_test.test_ext_data_sources.TestExtDataSources.test_verify_jdbc_table_properties,
expected jdbc jar url on S3 is:
s3a://impala-test-uswest2-2/test-warehouse/data-sources/jdbc-data-source.jar
{code:java}
query_test/test_ext_data_sources.py:70: in test_verify_jdbc_table_properties
assert properties['__IMPALA_DATA_SOURCE_LOCATION'] == \
E assert 's3a://impala...ta-source.jar' == 'hdfs://localh...ta-source.jar'
E -
s3a://impala-test-uswest2-2/test-warehouse/data-sources/jdbc-data-source.jar
E ? -- ^^^ ^^^^ ^^^^^^^ ^^
E + hdfs://localhost:20500/test-warehouse/data-sources/jdbc-data-source.jar
E ? +++ ^^^ ^^ ^ ^^^^
{code}
Stack trace of the test failure of
test_ext_data_sources.TestExtDataSources.test_jdbc_data_source
{code:java}
1012 05:26:05.270896 1181 GenericJdbcDatabaseAccessor.java:145]
c64d3ba0d79d1aec:076d0d8f00000001] Caught exception while trying to execute
query
Java exception follows:
java.sql.SQLException: Cannot create PoolableConnectionFactory (FATAL: database
"functional" does not exist)
at
org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:669)
at
org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:544)
at
org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:753)
at
org.apache.impala.extdatasource.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:137)
at
org.apache.impala.extdatasource.jdbc.JdbcDataSource.buildQueryAndExecute(JdbcDataSource.java:306)
at
org.apache.impala.extdatasource.jdbc.JdbcDataSource.open(JdbcDataSource.java:158)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.open(ExternalDataSourceExecutor.java:269)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.open(ExternalDataSourceExecutor.java:212)
Caused by: org.postgresql.util.PSQLException: FATAL: database "functional" does
not exist
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676)
at
org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2788)
at
org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:174)
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:290)
at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:247)
at org.postgresql.Driver.makeConnection(Driver.java:434)
at org.postgresql.Driver.connect(Driver.java:291)
at
org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:55)
at
org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:355)
at
org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:115)
at
org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:665)
... 7 more
E1012 05:26:05.271389 1181 ExternalDataSourceExecutor.java:253]
c64d3ba0d79d1aec:076d0d8f00000001] Error in data source
(path=/tmp/jdbc-data-source.6397.13.jar,
class=org.apache.impala.extdatasource.jdbc.JdbcDataSource, version=V1) close():
No error message returned by data source. Check the impalad log for more
information.
Java exception follows:
java.lang.IllegalArgumentException
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:131)
at
org.apache.impala.extdatasource.jdbc.JdbcDataSource.close(JdbcDataSource.java:211)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.close(ExternalDataSourceExecutor.java:289)
at
org.apache.impala.extdatasource.ExternalDataSourceExecutor.close(ExternalDataSourceExecutor.java:234)
{code}
> Support different schemes for jdbc driver url when creating external jdbc
> table
> -------------------------------------------------------------------------------
>
> Key: IMPALA-12470
> URL: https://issues.apache.org/jira/browse/IMPALA-12470
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
> Reporter: Wenzhe Zhou
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> jdbc driver url is specified as table property 'driver.url' when creating a
> external jdbc table, like the sample below:
> {code:java}
> CREATE TABLE test_postgres (
> id INT,
> string_col STRING)
> PRODUCED BY DATA SOURCE JdbcDataSource(
> '{"database.type":"POSTGRES",
> "jdbc.url":"jdbc:postgresql://localhost:5432/functional",
> "jdbc.driver":"org.postgresql.Driver",
> "driver.url":"hdfs://localhost:20500/test-warehouse/data-sources/jdbc-driver/postgresql-jdbc.jar",
> "dbcp.username":"hiveuser",
> "dbcp.password":"password",
> "table":"alltypes"}');
> {code}
> In the initial patch, we only support to download the driver jar files from
> hdfs. We need to support different storage type, like 'file://', 's3a://',
> etc.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
