[jira] [Comment Edited] (HIVE-21954) QTest: support for running qtests on various metastore DBs
[
https://issues.apache.org/jira/browse/HIVE-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965365#comment-16965365
]
László Bodor edited comment on HIVE-21954 at 11/9/19 8:00 AM:
--
[~kgyrtkirk], [~pvary]: could you please take a look? the patch contains the
following:
1. Database rule refactoring in standalone-metastore project
2. Qtest infrastructure: preparing qtest infra for being able to run any qtest
against any kind of metastores
3. A bunch of cleanups in the current qtest mess
With the patch, you can run the following (if your docker is set up properly,
meaning that you can run the metastore integration tests, already described in
standalone-metastore/DEV-README):
{code}
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=mssql
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=mysql
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=postgres
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=oracle
-Ditest.jdbc.jars=/path/to/your/god/damn/oracle/jdbc/driver/ojdbc6.jar
{code}
looking forward for comments: https://reviews.apache.org/r/71711/
and the data can be checked in metastore if you prevent it from cleaning up:
{code}
export QTEST_LEAVE_FILES=true
mvn ... -Dmetastore.itest.no.stop.container=true
{code}
was (Author: abstractdog):
[~kgyrtkirk], [~pvary]: could you please take a look? the patch contains the
following:
1. Database rule refactoring in standalone-metastore project
2. Qtest infrastructure: preparing qtest infra for being able to run any qtest
against any kind of metastores
3. A bunch of cleanups in the current qtest mess
With the patch, you can run the following (if your docker is set up properly,
meaning that you can run the metastore integration tests, already described in
standalone-metastore/DEV-README):
{code}
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=mssql
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=mysql
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=postgres
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=oracle
-Ditest.jdbc.jars=/path/to/your/god/damn/oracle/jdbc/driver/ojdbc6.jar
{code}
looking forward for comments: https://reviews.apache.org/r/71711/
> QTest: support for running qtests on various metastore DBs
> --
>
> Key: HIVE-21954
> URL: https://issues.apache.org/jira/browse/HIVE-21954
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21954.01.patch, HIVE-21954.02.patch,
> HIVE-21954.03.patch, HIVE-21954.03.patch, HIVE-21954.03.patch,
> HIVE-21954.04.patch, HIVE-21954.05.patch, HIVE-21954.07.patch,
> HIVE-21954.07.patch, HIVE-21954.08.patch, HIVE-21954.09.patch
>
>
> In HIVE-21940, a postgres metastore related issue has been fixed, and a local
> reproduction has been provided.
> {code}
> export QTEST_LEAVE_FILES=true
> docker kill metastore-test-postgres-install
> docker rm metastore-test-postgres-install
> cd standalone-metastore
> mvn verify -DskipITests=false -Dit.test=ITestPostgres#install -Dtest=nosuch
> -Dmetastore.itest.no.stop.container=true
> cd ..
> mvn test -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl itests/qtest
> -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
> -Dhive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore
> {code}
> The problem with this solution is that data/conf/hive-site.xml has to be
> edited manually. My proposal is to introduce a property
> (-Dmetastore.db=postgres), which can take care of the parameters on the fly.
> 2 supported solutions could be:
> 1. simple parameters: -Dmetastore.db=postgres
> In this case, tests depend on settings from ITestPostgres class (password,
> db, etc.)
> 2. verbose but flexible parameters: [see hive-site.xml HIVE-21940's repro
> patch|https://issues.apache.org/jira/secure/attachment/12973534/HIVE-21940.repro.patch]
>
> In the first implementation, I
[jira] [Comment Edited] (HIVE-21954) QTest: support for running qtests on various metastore DBs
[
https://issues.apache.org/jira/browse/HIVE-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965365#comment-16965365
]
László Bodor edited comment on HIVE-21954 at 11/2/19 1:34 PM:
--
[~kgyrtkirk], [~pvary]: could you please take a look? the patch contains the
following:
1. Database rule refactoring in standalone-metastore project
2. Qtest infrastructure: preparing qtest infra for being able to run any qtest
against any kind of metastores
3. A bunch of cleanups in the current qtest mess
With the patch, you can run the following (if your docker is set up properly,
meaning that you can run the metastore integration tests, already described in
standalone-metastore/DEV-README):
{code}
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=mssql
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=mysql
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=postgres
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=oracle
-Ditest.jdbc.jars=/path/to/your/god/damn/oracle/jdbc/driver/ojdbc6.jar
{code}
looking forward for comments: https://reviews.apache.org/r/71711/
was (Author: abstractdog):
[~kgyrtkirk], [~pvary]: could you please take a look? the patch contains the
following:
1. Database rule refactoring in standalone-metastore project
2. Qtest infrastructure: preparing qtest infra for being able to run any qtest
against any kind of metastores
3. A bunch of cleanups in the current qtest mess
With the patch, you can run the following (if your docker is set up properly,
meaning that you can run the metastore integration tests, already described in
standalone-metastore/DEV-README):
{code}
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=mssql
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=mysql
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=postgres
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dtest.metastore.db=oracle
-Ditest.jdbc.jars=/path/to/your/god/damn/oracle/jdbc/driver/ojdbc6.jar
{code}
> QTest: support for running qtests on various metastore DBs
> --
>
> Key: HIVE-21954
> URL: https://issues.apache.org/jira/browse/HIVE-21954
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21954.01.patch, HIVE-21954.02.patch,
> HIVE-21954.03.patch, HIVE-21954.03.patch, HIVE-21954.03.patch,
> HIVE-21954.04.patch, HIVE-21954.05.patch, HIVE-21954.07.patch,
> HIVE-21954.07.patch, HIVE-21954.08.patch
>
>
> In HIVE-21940, a postgres metastore related issue has been fixed, and a local
> reproduction has been provided.
> {code}
> export QTEST_LEAVE_FILES=true
> docker kill metastore-test-postgres-install
> docker rm metastore-test-postgres-install
> cd standalone-metastore
> mvn verify -DskipITests=false -Dit.test=ITestPostgres#install -Dtest=nosuch
> -Dmetastore.itest.no.stop.container=true
> cd ..
> mvn test -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl itests/qtest
> -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
> -Dhive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore
> {code}
> The problem with this solution is that data/conf/hive-site.xml has to be
> edited manually. My proposal is to introduce a property
> (-Dmetastore.db=postgres), which can take care of the parameters on the fly.
> 2 supported solutions could be:
> 1. simple parameters: -Dmetastore.db=postgres
> In this case, tests depend on settings from ITestPostgres class (password,
> db, etc.)
> 2. verbose but flexible parameters: [see hive-site.xml HIVE-21940's repro
> patch|https://issues.apache.org/jira/secure/attachment/12973534/HIVE-21940.repro.patch]
>
> In the first implementation, I would not start metastore db automatically
> (which is done be 'mvn verify ...'), but it's still under planning.
> In the long term, we should consider running this kind of tests in precommit
> phase, so maybe -Dmetastore.db=postgres could start metastore
