This is an automated email from the ASF dual-hosted git repository.
okumin pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hive-site.git
The following commit(s) were added to refs/heads/main by this push:
new cc176b9 HIVE-28845: Document for QFile Tests (#45) (Shohei Okumiya,
reviewed by Denys Kuzmenko, Butao Zhang, Zoltan Ratkai, Stamatis Zampetakis)
cc176b9 is described below
commit cc176b9981c2be369fe47910559e61b87c34b566
Author: Shohei Okumiya <[email protected]>
AuthorDate: Thu Apr 3 17:14:10 2025 +0900
HIVE-28845: Document for QFile Tests (#45) (Shohei Okumiya, reviewed by
Denys Kuzmenko, Butao Zhang, Zoltan Ratkai, Stamatis Zampetakis)
---
content/Development/qtest.md | 196 ++++++++++++++++++
content/docs/latest/developerguide_27362074.md | 130 +-----------
content/docs/latest/hivedeveloperfaq_27823747.md | 241 +----------------------
content/docs/latest/howtocontribute_27362107.md | 63 +-----
layouts/shortcodes/toc.html | 3 +
5 files changed, 208 insertions(+), 425 deletions(-)
diff --git a/content/Development/qtest.md b/content/Development/qtest.md
new file mode 100644
index 0000000..a2ef5d2
--- /dev/null
+++ b/content/Development/qtest.md
@@ -0,0 +1,196 @@
+---
+title: "Query File Test(qtest)"
+date: 2025-03-28
+draft: false
+aliases: [/qtest.html]
+---
+
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License. -->
+
+# Query File Test(qtest)
+
+Query File Test is a JUnit-based integration test suite for Apache Hive.
Developers write any SQL; the testing framework runs it and verifies the result
and output.
+
+{{< toc >}}
+
+## Tutorial: How to run a specific test case
+
+### Preparation
+
+You have to compile Hive's source codes ahead of time.
+
+```sh
+$ mvn clean install -Dmaven.javadoc.skip=true -DskipTests -Pitests
+```
+
+### Run a test case
+
+Let's try to run
[alter1.q](https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/alter1.q).
+
+```sh
+$ mvn test -Pitests -pl itests/qtest -Dtest=TestMiniLlapLocalCliDriver
-Dqfile=alter1.q
+```
+
+The test should successfully finish.
+
+```sh
+[INFO] Results:
+[INFO]
+[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
+```
+
+## Tutorial: How to add a new test case
+
+### Add a QFile
+
+A QFile includes a set of SQL statements that you want to test. Typically, we
should put a new file in `ql/src/test/queries/clientpositive`.
+
+Let's say you created the following file.
+
+```sh
+$ cat ql/src/test/queries/clientpositive/aaa.q
+SELECT 1;
+```
+
+### Generate a result file
+
+You can generate the expected output with `-Dtest.output.overwrite=true`.
+
+```sh
+$ mvn test -Pitests -pl itests/qtest -Dtest=TestMiniLlapLocalCliDriver
-Dtest.output.overwrite=true -Dqfile=aaa.q
+...
+$ cat ql/src/test/results/clientpositive/llap/aaa.q.out
+PREHOOK: query: SELECT 1
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+POSTHOOK: query: SELECT 1
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+1
+```
+
+### Verify the new result file
+
+You can ensure the generated result file is correct by rerunning the test case
without `-Dtest.output.overwrite=true`.
+
+```sh
+$ mvn test -Pitests -pl itests/qtest -Dtest=TestMiniLlapLocalCliDriver
-Dqfile=aaa.q
+```
+
+## Commandline options
+
+### Test options
+
+| Option | Description | Example |
+|-|-|-|
+| test | The class name of the test driver |
`-Dtest=TestMiniLlapLocalCliDriver` |
+| qfile | The name(s) of Query Files | `-Dqfile=alter1.q`,
`-Dqfile=alter1.q,alter2.q` |
+| qfile_regex | The pattern to list Query Files | `-Dqfile_regex=alter[0-9]` |
+| test.output.overwrite | Whether you want to (re)generate result files or not
| `-Dtest.output.overwrite=true` |
+| test.metastore.db | Which RDBMS to be used as a metastore backend | See [How
to use PostgreSQL/MySQL/Oracle as a backend database for Hive
Metastore](#how-to-use-postgresqlmysqloracle-as-a-backend-database-for-hive-metastore)
|
+
+### Test Iceberg, Accumulo, or Kudu
+
+Most test drivers are available in the `itest/qtest` project. However, there
are some exceptional ones.
+
+| Driver | Project |
+|-|-|
+| TestAccumuloCliDriver | itest/qtest-accumulo |
+| TestIcebergCliDriver | itest/qtest-iceberg |
+| TestIcebergLlapLocalCliDriver | itest/qtest-iceberg |
+| TestIcebergLlapLocalCompactorCliDriver | itest/qtest-iceberg |
+| TestIcebergNegativeCliDriver | itest/qtest-iceberg |
+| TestKuduCliDriver | itest/qtest-kudu |
+| TestKuduNegativeCliDriver | itest/qtest-kudu |
+
+When you use `TestIcebergLlapLocalCliDriver`, you have to specify `-pl
itest/qtest-iceberg`.
+
+```sh
+$ mvn test -Pitests -pl itests/qtest-iceberg
-Dtest=TestIcebergLlapLocalCliDriver -Dqfile_regex=iceberg_bucket_map_join_8
+```
+
+## QTestOptionHandler: pre/post-processor
+
+We extend JUnit by implementing
[QTestOptionHandlers](https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/ql/qoption/QTestOptionHandler.java),
which are custom pre-processors and post-processors. This section explains a
couple of typical processors.
+
+### Using test data
+
+Adding `--! qt:dataset:{table name}`,
[QTestDatasetHandler](https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/ql/dataset/QTestDatasetHandler.java)
automatically sets up a test table. You can find the table definitions
[here](https://github.com/apache/hive/tree/master/data/files/datasets).
+
+```sql
+--! qt:dataset:src
+SELECT * FROM src;
+```
+
+### Mask non-deterministic outputs
+
+Some test cases generate random results.
[QTestReplaceHandler](https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/ql/qoption/QTestReplaceHandler.java)
masks such a non-deterministic part. You can use it with a special comment
prefixed with `--! qt:replace:`.
+
+For example, the result of `CURRENT_DATE` changes every day. Using the
comment, the output will be `non-deterministic-output #Masked#`, which is
stable across executions.
+
+```sql
+--!
qt:replace:/(non-deterministic-output\s)[0-9]{4}-[0-9]{2}-[0-9]{2}/$1#Masked#/
+SELECT 'non-deterministic-output', CURRENT_DATE();
+```
+
+## Advanced
+
+### Locations of log files
+
+The Query File Test framework outputs log files in the following paths.
+
+- `itests/{qtest, qtest-accumulo, qtest-iceberg,
qtest-kudu}/target/surefire-reports`
+- From the root of the source tree: `find . -name hive.log`
+
+### Negative tests
+
+Negative drivers allow us to make sure that a test case fails expectedly. For
example, the query in
[strict_timestamp_to_numeric.q](https://github.com/apache/hive/blob/master/ql/src/test/queries/clientnegative/strict_timestamp_to_numeric.q)
must fail based on Hive’s specifications. We can use
`TestNegativeLlapLocalCliDriver`, `TestIcebergNegativeCliDriver`, and so on.
+
+```sh
+$ mvn -Pitests -pl itests/qtest test -Dtest=TestNegativeLlapLocalCliDriver
-Dqfile=strict_timestamp_to_numeric.q
+```
+
+### How to specify drivers
+
+We define the default mapping of Query Files and test drivers using
[testconfiguration.properties](https://github.com/apache/hive/blob/master/itests/src/test/resources/testconfiguration.properties)
and
[CliConfigs](https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java).
For example, `TestMiniLlapLocalCliDriver` is the default driver for query
files stored in `ql/src/test/queries/clientpositive`. [The hive-precommit
Jenkins [...]
+
+You can override the mapping through
[testconfiguration.properties](https://github.com/apache/hive/blob/master/itests/src/test/resources/testconfiguration.properties).
For example, if you want to test `ql/src/test/queries/clientpositive/aaa.q`
not by LLAP but by Tez, you must include the file name in `minitez.query.files`
and generate the result file with `-Dtest=TestMiniTezCliDriver`.
+
+In most cases, we should use `TestMiniLlapLocalCliDriver` for positive tests
and `TestNegativeLlapLocalCliDriver` for negative tests.
+
+### How to use PostgreSQL/MySQL/Oracle as a backend database for Hive Metastore
+
+To run a test with a specified DB, it is possible by adding the
"-Dtest.metastore.db" parameter like in the following commands:
+
+```sh
+$ mvn test -Pitests -pl itests/qtest -Dtest=TestCliDriver
-Dqfile=partition_params_postgres.q -Dtest.metastore.db=postgres
+$ mvn test -Pitests -pl itests/qtest -Dtest=TestCliDriver
-Dqfile=partition_params_postgres.q -Dtest.metastore.db=mssql
+$ mvn test -Pitests -pl itests/qtest -Dtest=TestCliDriver
-Dqfile=partition_params_postgres.q -Dtest.metastore.db=mysql
+$ mvn test -Pitests -pl itests/qtest -Dtest=TestCliDriver
-Dqfile=partition_params_postgres.q -Dtest.metastore.db=oracle
-Ditest.jdbc.jars=/path/to/your/god/damn/oracle/jdbc/driver/ojdbc6.jar
+```
+
+### Remote debug
+
+Remote debugging with Query File Test is a potent tool for debugging Hive.
With the following command, Query File Test listens to port 5005 and waits for
a debugger to be attached.
+
+```sh
+$ mvn -Pitests -pl itests/qtest -Dmaven.surefire.debug test
-Dtest=TestMiniLlapLocalCliDriver -Dqfile=alter1.q
+```
diff --git a/content/docs/latest/developerguide_27362074.md
b/content/docs/latest/developerguide_27362074.md
index 2d8343d..9e4b58f 100644
--- a/content/docs/latest/developerguide_27362074.md
+++ b/content/docs/latest/developerguide_27362074.md
@@ -5,44 +5,7 @@ date: 2024-12-12
# Apache Hive : DeveloperGuide
-# Developer Guide
-
-* [Developer Guide]({{< ref "#developer-guide" >}})
- + [Code Organization and a Brief Architecture]({{< ref
"#code-organization-and-a-brief-architecture" >}})
- - [Introduction]({{< ref "#introduction" >}})
- - [Hive SerDe]({{< ref "#hive-serde" >}})
- * [How to Write Your Own SerDe]({{< ref
"#how-to-write-your-own-serde" >}})
- * [ObjectInspector]({{< ref "#objectinspector" >}})
- * [Registration of Native SerDes]({{< ref
"#registration-of-native-serdes" >}})
- - [MetaStore]({{< ref "#metastore" >}})
- - [Query Processor]({{< ref "#query-processor" >}})
- * [Compiler]({{< ref "#compiler" >}})
- * [Parser]({{< ref "#parser" >}})
- * [TypeChecking]({{< ref "#typechecking" >}})
- * [Semantic Analysis]({{< ref "#semantic-analysis" >}})
- * [Plan generation]({{< ref "#plan-generation" >}})
- * [Task generation]({{< ref "#task-generation" >}})
- * [Execution Engine]({{< ref "#execution-engine" >}})
- * [Plan]({{< ref "#plan" >}})
- * [Operators]({{< ref "#operators" >}})
- * [UDFs and UDAFs]({{< ref "#udfs-and-udafs" >}})
- + [Compiling and Running Hive]({{< ref "#compiling-and-running-hive"
>}})
- - [Default Mode]({{< ref "#default-mode" >}})
- - [Advanced Mode]({{< ref "#advanced-mode" >}})
- - [Running Hive Without a Hadoop Cluster]({{< ref
"#running-hive-without-a-hadoop-cluster" >}})
- + [Unit tests and debugging]({{< ref "#unit-tests-and-debugging" >}})
- - [Layout of the unit tests]({{< ref
"#layout-of-the-unit-tests" >}})
- - [Running unit tests]({{< ref "#running-unit-tests" >}})
- - [Adding new unit tests]({{< ref "#adding-new-unit-tests" >}})
- - [Debugging Hive Code]({{< ref "#debugging-hive-code" >}})
- * [Debugging Client-Side Code]({{< ref
"#debugging-client-side-code" >}})
- * [Debugging Server-Side Code]({{< ref
"#debugging-server-side-code" >}})
- * [Debugging without Ant (Client and Server Side)]({{<
ref "#debugging-without-ant-client-and-server-side" >}})
- + [Pluggable interfaces]({{< ref "#pluggable-interfaces" >}})
- - [File Formats]({{< ref "#file-formats" >}})
- - [SerDe - how to add a new SerDe]({{< ref
"#serde---how-to-add-a-new-serde" >}})
- - [Map-Reduce Scripts]({{< ref "#map-reduce-scripts" >}})
- - [UDFs and UDAFs - how to add new UDFs and UDAFs]({{< ref
"#udfs-and-udafs---how-to-add-new-udfs-and-udafs" >}})
+{{< toc >}}
## Code Organization and a Brief Architecture
@@ -286,96 +249,7 @@ Then you can run '`build/dist/bin/hive`' and it will work
against your local fil
### Layout of the unit tests
-Hive uses [JUnit](http://junit.org/) for unit tests. Each of the 3 main
components of Hive have their unit test implementations in the corresponding
src/test directory e.g. trunk/metastore/src/test has all the unit tests for
metastore, trunk/serde/src/test has all the unit tests for serde and
trunk/ql/src/test has all the unit tests for the query processor. The metastore
and serde unit tests provide the TestCase implementations for JUnit. The query
processor tests on the other hand are g [...]
-
-* Test Queries:
- + queries/clientnegative - This directory contains the query files (.q
files) for the negative test cases. These are run through the CLI classes and
therefore test the entire query processor stack.
- + queries/clientpositive - This directory contains the query files (.q
files) for the positive test cases. Thesre are run through the CLI classes and
therefore test the entire query processor stack.
- + qureies/positive (Will be deprecated) - This directory contains the
query files (.q files) for the positive test cases for the compiler. These only
test the compiler and do not run the execution code.
- + queries/negative (Will be deprecated) - This directory contains the
query files (.q files) for the negative test cases for the compiler. These only
test the compiler and do not run the execution code.
-* Test Results:
- + results/clientnegative - The expected results from the queries in
queries/clientnegative.
- + results/clientpositive - The expected results from the queries in
queries/clientpositive.
- + results/compiler/errors - The expected results from the queries in
queries/negative.
- + results/compiler/parse - The expected Abstract Syntax Tree output for
the queries in queries/positive.
- + results/compiler/plan - The expected query plans for the queries in
queries/positive.
-* Velocity Templates to Generate the Tests:
- + templates/TestCliDriver.vm - Generates the tests from
queries/clientpositive.
- + templates/TestNegativeCliDriver.vm - Generates the tests from
queries/clientnegative.
- + templates/TestParse.vm - Generates the tests from queries/positive.
- + templates/TestParseNegative.vm - Generates the tests from
queries/negative.
-
-### Running unit tests
-
-Ant to Maven
-
-As of version [0.13](https://issues.apache.org/jira/browse/HIVE-5107) Hive
uses Maven instead of Ant for its build. The following instructions are not up
to date.
-
-See the [Hive Developer FAQ]({{< ref "#hive-developer-faq" >}}) for updated
instructions.
-
-Run all tests:
-
-```
-ant package test
-
-```
-
-Run all positive test queries:
-
-```
-ant test -Dtestcase=TestCliDriver
-
-```
-
-Run a specific positive test query:
-
-```
-ant test -Dtestcase=TestCliDriver -Dqfile=groupby1.q
-
-```
-
-The above test produces the following files:
-
-* `build/ql/test/TEST-org.apache.hadoop.hive.cli.TestCliDriver.txt` - Log
output for the test. This can be helpful when examining test failures.
-* `build/ql/test/logs/groupby1.q.out` - Actual query result for the test. This
result is compared to the expected result as part of the test.
-
-Run the set of unit tests matching a regex, e.g. partition_wise_fileformat
tests 10-16:
-
-```
-ant test -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat1[0-6]
-
-```
-
-Note that this option matches against the basename of the test without the .q
suffix.
-
-Apparently the Hive tests do not run successfully after a clean unless you run
`ant package` first. Not sure why build.xml doesn't encode this dependency.
-
-### Adding new unit tests
-
-Ant to Maven
-
-As of version [0.13](https://issues.apache.org/jira/browse/HIVE-5107) Hive
uses Maven instead of Ant for its build. The following instructions are not up
to date.
-
-See the [Hive Developer FAQ]({{< ref "#hive-developer-faq" >}}) for updated
instructions. See also [Tips for Adding New Tests in Hive]({{< ref
"tipsforaddingnewtests_27362060" >}}) and [How to Contribute: Add a Unit
Test]({{< ref "#how-to-contribute:-add-a-unit-test" >}}).
-
-First, write a new myname.q in ql/src/test/queries/clientpositive.
-
-Then, run the test with the query and overwrite the result (useful when you
add a new test).
-
-```
-ant test -Dtestcase=TestCliDriver -Dqfile=myname.q -Doverwrite=true
-
-```
-
-Then we can create a patch by:
-
-```
-svn add ql/src/test/queries/clientpositive/myname.q
ql/src/test/results/clientpositive/myname.q.out
-svn diff > patch.txt
-
-```
-
-Similarly, to add negative client tests, write a new query input file in
ql/src/test/queries/clientnegative and run the same command, this time
specifying the testcase name as TestNegativeCliDriver instead of TestCliDriver.
Note that for negative client tests, the output file if created using the
overwrite flag can be be found in the directory
ql/src/test/results/clientnegative.
+Hive uses [JUnit](http://junit.org/) for unit tests. Each of the 3 main
components of Hive have their unit test implementations in the corresponding
src/test directory e.g. trunk/metastore/src/test has all the unit tests for
metastore, trunk/serde/src/test has all the unit tests for serde and
trunk/ql/src/test has all the unit tests for the query processor. The metastore
and serde unit tests provide the TestCase implementations for JUnit.
### Debugging Hive Code
diff --git a/content/docs/latest/hivedeveloperfaq_27823747.md
b/content/docs/latest/hivedeveloperfaq_27823747.md
index 439ed20..f162b18 100644
--- a/content/docs/latest/hivedeveloperfaq_27823747.md
+++ b/content/docs/latest/hivedeveloperfaq_27823747.md
@@ -5,46 +5,7 @@ date: 2024-12-12
# Apache Hive : HiveDeveloperFAQ
-# Hive Developer FAQ
-
-* [Hive Developer FAQ]({{< ref "#hive-developer-faq" >}})
- + [Developing]({{< ref "#developing" >}})
- - [How do I move some files?]({{< ref
"#how-do-i-move-some-files" >}})
- - [How do I add a new MiniDriver test?]({{< ref
"#how-do-i-add-a-new-minidriver-test" >}})
- + [Building]({{< ref "#building" >}})
- - [Maven settings]({{< ref "#maven-settings" >}})
- - [How to build all source?]({{< ref "#how-to-build-all-source"
>}})
- - [How do I import into Eclipse?]({{< ref
"#how-do-i-import-into-eclipse" >}})
- - [How to generate tarball?]({{< ref "#how-to-generate-tarball"
>}})
- - [How to generate protobuf code?]({{< ref
"#how-to-generate-protobuf-code" >}})
- - [How to generate Thrift code?]({{< ref
"#how-to-generate-thrift-code" >}})
- - [How to run findbugs after a change?]({{< ref
"#how-to-run-findbugs-after-a-change?" >}})
- - [How to compile ODBC?]({{< ref "#how-to-compile-odbc" >}})
- - [How do I publish Hive artifacts to my local Maven
repository?]({{< ref
"#how-do-i-publish-hive-artifacts-to-my-local-maven-repository" >}})
- + [Testing]({{< ref "#testing" >}})
- - [How do I run precommit tests on a patch?]({{< ref
"#how-do-i-run-precommit-tests-on-a-patch" >}})
- - [How do I rerun precommit tests over the same patch?]({{< ref
"#how-do-i-rerun-precommit-tests-over-the-same-patch" >}})
- - [How do I run a single test?]({{< ref
"#how-do-i-run-a-single-test" >}})
- - [How do I run all of the unit tests?]({{< ref
"#how-do-i-run-all-of-the-unit-tests" >}})
- - [How do I run all of the unit tests except for a certain few
tests?]({{< ref
"#how-do-i-run-all-of-the-unit-tests-except-for-a-certain-few-tests" >}})
- - [How do I run the clientpositive/clientnegative unit
tests?]({{< ref "#how-do-i-run-the-clientpositiveclientnegative-unit-tests" >}})
- - [How do I run with Postgre/MySQL/Oracle?]({{< ref
"#how-do-i-run-with-postgremysqloracle" >}})
- - [How do I remote debug a qtest?]({{< ref
"#how-do-i-remote-debug-a-qtest" >}})
- - [How do I modify the init script when testing?]({{< ref
"#how-do-i-modify-the-init-script-when-testing" >}})
- - [How do I update the output of a CliDriver testcase?]({{< ref
"#how-do-i-update-the-output-of-a-clidriver-testcase" >}})
- - [How do I update the results of many test cases?]({{< ref
"#how-do-i-update-the-results-of-many-test-cases" >}})
- - [Where is the log output of a test?]({{< ref
"#where-is-the-log-output-of-a-test" >}})
- - [How do I add a test case?]({{< ref
"#how-do-i-add-a-test-case" >}})
- - [Why isn't the itests pom connected to the root pom?]({{< ref
"#why-isnt-the-itests-pom-connected-to-the-root-pom" >}})
- - [Why does a test fail with a NullPointerException in
MiniDFSCluster?]({{< ref
"#why-does-a-test-fail-with-a-nullpointerexception-in-minidfscluster" >}})
- - [Why do Spark unit tests fail with a SecurityException?]({{<
ref "#why-do-spark-unit-tests-fail-with-a-securityexception" >}})
- + [Debugging]({{< ref "#debugging" >}})
- - [How do I debug into a single test in Eclipse?]({{< ref
"#how-do-i-debug-into-a-single-test-in-eclipse?" >}})
- - [How do I debug my queries in Hive?]({{< ref
"#how-do-i-debug-my-queries-in-hive" >}})
-
-Maven
-
-Run the test and generate the output file using the appropriate `-Dtest`Hive
is using [Maven]({{< ref "#maven" >}}) as its build tool. Versions prior to
0.13 were using Ant.
+{{< toc >}}
## Developing
@@ -230,16 +191,11 @@ For general information, see [Unit Tests and
Debugging]({{< ref "#unit-tests-and
### How do I run precommit tests on a patch?
-Hive precommit testing is triggered automatically when a file is uploaded to
the JIRA ticket:
-
-1. Attach the patch file to a JIRA ticket: in the ticket's "More" tab, select
"Attach Files" and use "Choose File" to upload the file, then add a descriptive
comment.
-2. Put the patch in the review queue: click the "Submit Patch" button. The
button name will change to "Cancel Patch" and the ticket status will change to
Patch Available.
-
-See [Hive PreCommit Patch Testing]({{< ref
"hive-precommit-patch-testing_33295252" >}}) for more detail.
+A Jenkins job will start when you create a pull request on GitHub.
### How do I rerun precommit tests over the same patch?
-For patch updates, our convention is to number them like HIVE-1856.1.patch,
HIVE-1856.2.patch, etc. And then click the "Submit Patch" button again when a
new one is uploaded; this makes sure it gets back into the review queue.
+You can ask a committer to rerun it, or do `git commit --amend` and push it.
### How do I run a single test?
@@ -273,7 +229,6 @@ For more usage see the documentation for the [Maven
Surefire Plugin](http://mave
mvn test
cd itests
mvn test
-
```
Note that you need to have previously built and installed the jars:
@@ -282,11 +237,8 @@ Note that you need to have previously built and installed
the jars:
mvn clean install -DskipTests
cd itests
mvn clean install -DskipTests
-
```
-
-
### How do I run all of the unit tests except for a certain few tests?
Similar to running all tests, but define test.excludes.additional to specify a
test/pattern to exclude from the test run. For example the following will run
all tests except for the CliDriver tests:
@@ -298,163 +250,7 @@ mvn test
-Dtest.excludes.additional='**/Test*CliDriver.java'
### How do I run the clientpositive/clientnegative unit tests?
-All of the below require that you have previously run `ant package`.
-
-To run clientpositive tests
-
-```
-cd itests/qtest
-mvn test -Dtest=TestCliDriver
-```
-
-To run a single clientnegative test alter1.q
-
-```
-cd itests/qtest
-mvn test -Dtest=TestNegativeCliDriver -Dqfile=alter1.q
-```
-
-To run all of the clientpositive tests that match a regex, for example the
partition_wise_fileformat tests
-
-```
-cd itests/qtest
-mvn test -Dtest=TestCliDriver -Dqfile_regex=partition_wise_fileformat.*
-
-# Alternatively, you can specify comma separated list with "-Dqfile" argument
-mvn test -Dtest=TestMiniLlapLocalCliDriver
-Dqfile='vectorization_0.q,vectorization_17.q,vectorization_8.q'
-```
-
-To run a single contrib test alter1.q and overwrite the result file
-
-```
-cd itests/qtest
-mvn test -Dtest=TestContribCliDriver -Dqfile=alter1.q
-Dtest.output.overwrite=true
-```
-
-### How do I run with Postgre/MySQL/Oracle?
-
-To run test test with a specified DB it is possible by adding
"-Dtest.metastore.db" parameter like in the following commands:
-
-```
-mvn test -Pitests -pl itests/qtest -Dtest=TestCliDriver
-Dqfile=partition_params_postgres.q -Dtest.metastore.db=postgres
-
-mvn test -Pitests -pl itests/qtest -Dtest=TestCliDriver
-Dqfile=partition_params_postgres.q -Dtest.metastore.db=mssql
-mvn test -Pitests -pl itests/qtest -Dtest=TestCliDriver
-Dqfile=partition_params_postgres.q -Dtest.metastore.db=mysql
-mvn test -Pitests -pl itests/qtest -Dtest=TestCliDriver
-Dqfile=partition_params_postgres.q -Dtest.metastore.db=oracle
-Ditest.jdbc.jars=/path/to/your/god/damn/oracle/jdbc/driver/ojdbc6.jar
-```
-
-Without specifying -Dqfile it will run all .q files .
-
-### How do I remote debug a qtest?
-
-```
-cd itests/qtest
-mvn -Dmaven.surefire.debug="-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 -Xnoagent
-Djava.compiler=NONE" test -Dtest=TestCliDriver -Dqfile=<test>.q
-```
-
-### How do I modify the init script when testing?
-
-The option to skip the init script or supply a custom init script was added in
Hive 2.0 (see [HIVE-11538](https://issues.apache.org/jira/browse/HIVE-11538)).
-
-To skip initialization:
-
-```
-mvn test -Dtest=TestCliDriver -Phadoop-2 -Dqfile=test_to_run.q -DinitScript=
-```
-
-To supply a custom script:
-
-```
-mvn test -Dtest=TestCliDriver -Phadoop-2 -Dtest.output.overwrite=true
-Dqfile=test_to_run.q -DinitScript=custom_script.sql
-```
-
-### How do I update the output of a CliDriver testcase?
-
-```
-cd itests/qtest
-mvn test -Dtest=TestCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true
-
-```
-
-
-
-### How do I update the results of many test cases?
-
-Assume that you have a file like below which you'd like to re-generate output
files for. Such a file could be created by copying the output from the
precommit tests.
-
-```
-head -2 /tmp/failed-TestCliDriver-file-tests
-org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
-org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
-```
-
-You can re-generate all those output files in batches of 20 with the command
below
-
-```
-egrep 'TestCliDriver' /tmp/failed-TestCliDriver-file-tests | perl -pe
's@.*testCliDriver_@@g' | awk '{print $1 ".q"}' | xargs -n 30 | perl -pe 's@
@,@g' | xargs -I{} mvn test -Dtest=TestCliDriver -Dtest.output.overwrite=true
-Dqfile={}
-```
-
-To do the same from the output of a precommit result, with multiple drivers,
you can do
-
-```
-import re
-from itertools import groupby
-s = """
-org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=231)
-org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
-org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
-"""
-PAT = re.compile("org.apache.hadoop.hive.cli.([^\.]*).*\[([^\]]*).*")
-l = [PAT.match(x.strip()) for x in s.split("\n") if x.strip()]
-for driver,q in groupby(sorted([a.groups() for a in l if a]), key=lambda
a:a[0]):
- print """mvn clean test -Dtest=%s '-Dqfile=%s'
-Dtest.output.overwrite=true""" % (driver, ",".join(["%s.q" % a[1] for a in q]))
-
-```
-
-### Where is the log output of a test?
-
-Logs are put in a couple locations:
-
-* ```
-From the root of the source tree: find . -name hive.log
-```
-* ```
-/tmp/$USER/ (Linux) or $TMPDIR/$USER/ (MacOS)
-```
-
-See [Hive Logging]({{< ref "#hive-logging" >}}) for details about log files,
including alternative configurations.
-
-### How do I add a test case?
-
-First, add the test case to the qfile test suite:
-
-* Copy the test to a new file under
`ql/src/test/queries/clientpositive/<filename>.q` (or `/clientnegative` if it
is a negative test).
- + If the new test creates any table, view, function, etc., make sure
that the name is unique across tests. For instance, name a table in the test
file `foo.q`, `foo_t1` instead of simply `t1`. This will help reduce flakiness
in the test runs, since Jenkins will run tests and batches, and currently it
does not restore to former state after running each of the q files.
- + If there is any interaction with file system, use unique folders for
the test to avoid any collision with other tests.
-* Add the `<filename.q>` to
`itests/src/test/resources/testconfiguration.properties` to the appropriate
variable (ex. `minimr.query.files`).
-
-Next, generate the golden (output) file the first time you run the test case:
-
-* Compile the Hive source from the top level Hive directory:
-
-```
-mvn clean install -DskipTests
-```
-* Compile the itests:
-
-```
-cd itests
-mvn clean install -DskipTests
-```
-* Run the test and generate the output file using the appropriate `-Dtest`
(ex. `TestCliDriver`; see `itests/qtest/pom.xml` for the names of other test
suites):
-
-```
-cd qtest
-mvn test -Dtest=TestCliDriver -Dqfile=<filename>.q -Dtest.output.overwrite=true
-```
-* Add your output file to
`ql/src/test/results/clientpositive/<filename>.q.out` (or `/clientnegative` if
it is a negative test).
-
-With the above steps, you can [create a patch]({{< ref "#create-a-patch" >}})
which has a `.java` file, a `.q` file and a `.q.out` file.
+[See this page](/development/qtest/).
### Why isn't the itests pom connected to the root pom?
@@ -484,22 +280,6 @@ java.lang.NullPointerException: null
```
-### Why do Spark unit tests fail with a SecurityException?
-
-If you get the following errors in the unit tests:
-
-```
-2016-01-07T09:51:49,365 ERROR [Driver[]]: spark.SparkContext
(Logging.scala:logError(96)) - Error initializing SparkContext.
-java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s signer
information does not match signer information of other classes in the same
package
-
-```
-It happens because there are two conflicting versions of the same classes
"javax.servlet:servlet-api" and "org.eclipse.jetty.orbit:javax-servlet". Spark
requires the eclipse version, but most tools including Hadoop and Jetty depend
on the javax one. To avoid this problem, we need to exclude the javax version
everywhere it comes up. Fortunately, maven has a tool to do that with:
-
-```
-mvn dependency:tree -Dverbose
-```
-which prints out the dependency tree. Go to each directory with the failing
unit tests and search the dependency tree for "javax.servlet:servlet-api" and
use exclusions in the pom.xml to remove it. See
[HIVE-12783](https://issues.apache.org/jira/browse/HIVE-12783) for an example.
-
## Debugging
### How do I debug into a single test in Eclipse?
@@ -549,16 +329,3 @@ SET mapreduce.framework.name=local
```
At this point, attach the remote debugger as mentioned before to start
debugging your queries.
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/content/docs/latest/howtocontribute_27362107.md
b/content/docs/latest/howtocontribute_27362107.md
index 5def79d..46910c8 100644
--- a/content/docs/latest/howtocontribute_27362107.md
+++ b/content/docs/latest/howtocontribute_27362107.md
@@ -9,29 +9,7 @@ date: 2024-12-12
This page describes the mechanics of *how* to contribute software to Apache
Hive. For ideas about *what* you might contribute, please see open tickets in
[Jira](https://issues.apache.org/jira/browse/HIVE).
-* [How to Contribute to Apache Hive]({{< ref
"#how-to-contribute-to-apache-hive" >}})
- + [Getting the Source Code]({{< ref "#getting-the-source-code" >}})
- + [Becoming a Contributor]({{< ref "#becoming-a-contributor" >}})
- + [Making Changes]({{< ref "#making-changes" >}})
- - [Coding Conventions]({{< ref "#coding-conventions" >}})
- - [Understanding Maven]({{< ref "#understanding-maven" >}})
- - [Understanding Hive Branches]({{< ref
"#understanding-hive-branches" >}})
- - [Hadoop Dependencies]({{< ref "#hadoop-dependencies" >}})
- * [branch-1]({{< ref "#branch-1" >}})
- * [branch-2]({{< ref "#branch-2" >}})
- - [Unit Tests]({{< ref "#unit-tests" >}})
- - [Add a Unit Test]({{< ref "#add-a-unit-test" >}})
- * [Java Unit Test]({{< ref "#java-unit-test" >}})
- * [Query Unit Test]({{< ref "#query-unit-test" >}})
- * [Beeline Query Unit Test]({{< ref
"#beeline-query-unit-test" >}})
- - [Debugging]({{< ref "#debugging" >}})
- - [Submitting a PR]({{< ref "#submitting-a-pr" >}})
- - [Fetching a PR from Github]({{< ref
"#fetching-a-pr-from-github" >}})
- + [Contributing Your Work]({{< ref "#contributing-your-work" >}})
- + [JIRA]({{< ref "#jira" >}})
- - [Guidelines]({{< ref "#guidelines" >}})
- + [Generating Thrift Code]({{< ref "#generating-thrift-code" >}})
- + [See Also]({{< ref "#see-also" >}})
+{{< toc >}}
## Getting the Source Code
@@ -163,44 +141,9 @@ To test a particular component of Hive:
* Add a new class (name must start with `Test`) in the component's
`*/src/test/java` directory.
* To test only the new testcase, run `mvn test -Dtest=TestAbc` (where
`TestAbc` is the name of the new class), which will be faster than `mvn test`
which tests all testcases.
-#### Query Unit Test
+#### Query File Test(qtest)
-If the new feature can be tested using a Hive query in the command line, we
just need to add a new `*.q` file and a new `*.q.out` file.
-
-If the feature is added in `ql` (query language):
-
-* Add a new `XXXXXX.q` file in `ql/src/test/queries/clientpositive`.
(Optionally, add a new `XXXXXX.q` file for a query that is expected to fail in
`ql/src/test/queries/clientnegative`.)
-* Run `mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=XXXXXX.q
-Dtest.output.overwrite=true`. This will generate a new `XXXXXX.q.out` file in
`ql/src/test/results/clientpositive`.
- + If you want to run multiple .q files in the test run, you can specify
comma separated .q files, for example `-Dqfile="X1.q,X2.q"`. You can also
specify a Java regex, for example `-Dqfile_regex='join.*'`. (Note that it takes
Java regex, i.e., `'join.*`'`` and not `'join*'`.) The regex match first
removes the `.q` from the file name before matching regex, so specifying
`join*.q` will not work.
-
-If the feature is added in `contrib`:
-
-* Do the steps above, replacing `ql` with `contrib`, and `TestCliDriver` with
`TestContribCliDriver`.
-
-See the FAQ "[How do I add a test case?]({{< ref "#how-do-i-add-a-test-case?"
>}})" for more details.
-
-#### Beeline Query Unit Test
-
-Legacy query test Drivers (all of them except TestBeeLineDriver) uses HiveCli
to run the tests. TestBeeLineDriver runs the tests using the [Beeline
client]({{< ref "#beeline-client" >}}). Creates a specific database for them,
so the tests can run parallel. Running the tests you have the following
configuration options:
-
-* `-Dqfile=XXXXXX.q` - To run one or more specific query file tests. For the
exact format, check the Query Unit Test paragraph. If not provided only those
query files from `ql/src/test/queries/clientpositive` directory will be run
which are mentioned in `itests/src/test/resources/testconfiguration.properties`
in the `beeline.positive.include` parameter.
-* `-Dtest.output.overwrite=true` - This will rewrite the output of the q.out
files in `ql/src/test/results/clientpositive/beeline`. The default value is
false, and it will check the current output against the golden files
-* `-Dtest.beeline.compare.portable` - If this parameter is true, the generated
and the golden query output files will be filtered before comparing them. This
way the existing query tests can be run against different configurations using
the same golden output files. The result of the following commands will be
filtered out from the output files: EXPLAIN, DESCRIBE, DESCRIBE EXTENDED,
DESCRIBE FORMATTED, SHOW TABLES, SHOW FORMATTED INDEXES and SHOW DATABASES.
-The default value is `false`.
-* `-Djunit.parallel.threads=1` - The number of the parallel threads running
the tests. The default is `1`. There were some flakiness caused by
parallelization
-* `-Djunit.parallel.timeout=10` - The tests are terminated after the given
timeout. The parameter is set in minutes and the default is 10 minutes. (As of
[HIVE 3.0.0](https://issues.apache.org/jira/browse/HIVE-17072).)
-* The BeeLine tests could run against an existing cluster. Or if not provided,
then against a MiniHS2 cluster created during the tests.
- + `-Dtest.beeline.url` - The jdbc url which should be used to connect
to the existing cluster. If not set then a MiniHS2 cluster will be created
instead.
- + `-Dtest.beeline.user` - The user which should be used to connect to
the cluster. If not set `"user"` will be used.
- + `-Dtest.beeline.password` - The password which should be used to
connect to the cluster. If not set `"password"` will be used.
- + `-Dtest.data.dir` - The test data directory on the cluster. If not
set `<HIVEROOT>/data/files` will be used.
- + `-Dtest.results.dir` - The test results directory to compare against.
If not set the default configuration will be used.
- + `-Dtest.init.script` - The test init script. If not set the default
configuration will be used.
- + `-Dtest.beeline.shared.database` - If true, then the default database
will be used, otherwise a test-specific database will be created for every run.
The default value is false.
-
-### Debugging
-
-Please see [Debugging Hive code]({{< ref "#debugging-hive-code" >}}) in
Development Guide.
+[You can run end-to-end integration tests using LLAP, Tez, Iceberg,
etc.](/development/qtest/)
### Submitting a PR
diff --git a/layouts/shortcodes/toc.html b/layouts/shortcodes/toc.html
new file mode 100644
index 0000000..1a98f71
--- /dev/null
+++ b/layouts/shortcodes/toc.html
@@ -0,0 +1,3 @@
+<aside class="table-of-contents">
+ {{ .Page.TableOfContents }}
+</aside>
