[
https://issues.apache.org/jira/browse/HIVE-29079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora updated HIVE-29079:
---------------------------------
Description:
The SHOW COMPACTIONS command got extended with filtering in HIVE-13353. The
implementation supports filtering by partition name as well. This worked fine
after this change.
However in the scope of HIVE-27481 this part of the code got refactored a bit,
and unfortunately broke the filtering by partition.
How to reproduce:
* Create an ACID table with partitions and create some entries to the
compaction queue.
For example I have the following compactions for the table acid_3 which has 3
partitions:
{code:java}
0: jdbc:hive2://localhost:10000> show compactions show_compact_db.acid_3;
+---------------+------------------+----------+-----------+------------+---------------------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
| compactionid | dbname | tabname | partname | type |
state | workerhost | workerid | enqueuetime | starttime | duration
| hadoopjobid | errormessage | initiatorhost | initiatorid | poolname |
txnid | nexttxnid | committime | hightestwriteid |
+---------------+------------------+----------+-----------+------------+---------------------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
| 10 | show_compact_db | acid_3 | 2020 | MAJOR |
initiated | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool1
| 1113 | 1112 | 0 | --- |
| 11 | show_compact_db | acid_3 | 2020 | MAJOR |
succeeded | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool2
| 1113 | 1112 | 0 | --- |
| 12 | show_compact_db | acid_3 | 2020 | MINOR |
succeeded | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool1
| 1113 | 1112 | 0 | --- |
| 13 | show_compact_db | acid_3 | 2020 | REBALANCE |
succeeded | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool2
| 1113 | 1112 | 0 | --- |
| 14 | show_compact_db | acid_3 | 2021 | REBALANCE |
succeeded | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool3
| 1113 | 1112 | 0 | --- |
| 15 | show_compact_db | acid_3 | 2021 | MAJOR |
succeeded | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool3
| 1113 | 1112 | 0 | --- |
| 16 | show_compact_db | acid_3 | 2022 | MAJOR |
working | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool1
| 1113 | 1112 | 0 | --- |
| 17 | show_compact_db | acid_3 | 2022 | MINOR |
working | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool1
| 1113 | 1112 | 0 | --- |
| 18 | show_compact_db | acid_3 | 2022 | MAJOR |
succeeded | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool1
| 1113 | 1112 | 0 | --- |
| 19 | show_compact_db | acid_3 | 2022 | MAJOR | ready
for cleaning | 1234 | 1234 | 123456 | 123480 | ---
| None | --- | --- | --- | pool1 |
1113 | 1112 | 0 | --- |
+---------------+------------------+----------+-----------+------------+---------------------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
{code}
* Run the SHOW COMPACTIONS command with filtering on partition. The result is:
{code:java}
0: jdbc:hive2://localhost:10000> show compactions show_compact_db.acid_3
partition(year="2020");
+---------------+---------+----------+-----------+-------+--------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
| compactionid | dbname | tabname | partname | type | state | workerhost
| workerid | enqueuetime | starttime | duration | hadoopjobid |
errormessage | initiatorhost | initiatorid | poolname | txnid | nexttxnid
| committime | hightestwriteid |
+---------------+---------+----------+-----------+-------+--------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
+---------------+---------+----------+-----------+-------+--------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
{code}
was:
The SHOW COMPACTIONS command got extended with filtering in HIVE-13353. The
implementation supports filtering by partition name as well. This worked fine
after this change.
However in the scope of HIVE-27481 this part of the code got refactored a bit,
and unfortunately broke the filtering by partition.
How to reproduce:
* Create an ACID table with partitions and create some entries to the
compaction queue.
For example I have the following compactions for the table acid_3 which has 3
partitions:
{code:java}
0: jdbc:hive2://localhost:10000> show compactions show_compact_db.acid_3;
+---------------+------------------+----------+-----------+------------+---------------------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
| compactionid | dbname | tabname | partname | type |
state | workerhost | workerid | enqueuetime | starttime | duration
| hadoopjobid | errormessage | initiatorhost | initiatorid | poolname |
txnid | nexttxnid | committime | hightestwriteid |
+---------------+------------------+----------+-----------+------------+---------------------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
| 10 | show_compact_db | acid_3 | 2020 | MAJOR |
initiated | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool1
| 1113 | 1112 | 0 | --- |
| 11 | show_compact_db | acid_3 | 2020 | MAJOR |
succeeded | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool2
| 1113 | 1112 | 0 | --- |
| 12 | show_compact_db | acid_3 | 2020 | MINOR |
succeeded | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool1
| 1113 | 1112 | 0 | --- |
| 13 | show_compact_db | acid_3 | 2020 | REBALANCE |
succeeded | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool2
| 1113 | 1112 | 0 | --- |
| 14 | show_compact_db | acid_3 | 2021 | REBALANCE |
succeeded | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool3
| 1113 | 1112 | 0 | --- |
| 15 | show_compact_db | acid_3 | 2021 | MAJOR |
succeeded | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool3
| 1113 | 1112 | 0 | --- |
| 16 | show_compact_db | acid_3 | 2022 | MAJOR |
working | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool1
| 1113 | 1112 | 0 | --- |
| 17 | show_compact_db | acid_3 | 2022 | MINOR |
working | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool1
| 1113 | 1112 | 0 | --- |
| 18 | show_compact_db | acid_3 | 2022 | MAJOR |
succeeded | 1234 | 1234 | 123456 | 123480 |
--- | None | --- | --- | --- | pool1
| 1113 | 1112 | 0 | --- |
| 19 | show_compact_db | acid_3 | 2022 | MAJOR | ready
for cleaning | 1234 | 1234 | 123456 | 123480 | ---
| None | --- | --- | --- | pool1 |
1113 | 1112 | 0 | --- |
+---------------+------------------+----------+-----------+------------+---------------------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
{code}
* Run the SHOW COMPACTIONS command with filtering on partition. The result is:
{code:java}
0: jdbc:hive2://localhost:10000> show compactions show_compact_db.acid_3
partition(year=2020);
+---------------+---------+----------+-----------+-------+--------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
| compactionid | dbname | tabname | partname | type | state | workerhost
| workerid | enqueuetime | starttime | duration | hadoopjobid |
errormessage | initiatorhost | initiatorid | poolname | txnid | nexttxnid
| committime | hightestwriteid |
+---------------+---------+----------+-----------+-------+--------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
+---------------+---------+----------+-----------+-------+--------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
{code}
> Filtering by partitions in the SHOW COMPACTIONS command is broken
> -----------------------------------------------------------------
>
> Key: HIVE-29079
> URL: https://issues.apache.org/jira/browse/HIVE-29079
> Project: Hive
> Issue Type: Bug
> Affects Versions: 4.0.1
> Reporter: Marta Kuczora
> Priority: Major
>
> The SHOW COMPACTIONS command got extended with filtering in HIVE-13353. The
> implementation supports filtering by partition name as well. This worked fine
> after this change.
> However in the scope of HIVE-27481 this part of the code got refactored a
> bit, and unfortunately broke the filtering by partition.
> How to reproduce:
> * Create an ACID table with partitions and create some entries to the
> compaction queue.
> For example I have the following compactions for the table acid_3 which has 3
> partitions:
> {code:java}
> 0: jdbc:hive2://localhost:10000> show compactions show_compact_db.acid_3;
> +---------------+------------------+----------+-----------+------------+---------------------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
> | compactionid | dbname | tabname | partname | type |
> state | workerhost | workerid | enqueuetime | starttime |
> duration | hadoopjobid | errormessage | initiatorhost | initiatorid |
> poolname | txnid | nexttxnid | committime | hightestwriteid |
> +---------------+------------------+----------+-----------+------------+---------------------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
> | 10 | show_compact_db | acid_3 | 2020 | MAJOR |
> initiated | 1234 | 1234 | 123456 | 123480 |
> --- | None | --- | --- | --- |
> pool1 | 1113 | 1112 | 0 | --- |
> | 11 | show_compact_db | acid_3 | 2020 | MAJOR |
> succeeded | 1234 | 1234 | 123456 | 123480 |
> --- | None | --- | --- | --- |
> pool2 | 1113 | 1112 | 0 | --- |
> | 12 | show_compact_db | acid_3 | 2020 | MINOR |
> succeeded | 1234 | 1234 | 123456 | 123480 |
> --- | None | --- | --- | --- |
> pool1 | 1113 | 1112 | 0 | --- |
> | 13 | show_compact_db | acid_3 | 2020 | REBALANCE |
> succeeded | 1234 | 1234 | 123456 | 123480 |
> --- | None | --- | --- | --- |
> pool2 | 1113 | 1112 | 0 | --- |
> | 14 | show_compact_db | acid_3 | 2021 | REBALANCE |
> succeeded | 1234 | 1234 | 123456 | 123480 |
> --- | None | --- | --- | --- |
> pool3 | 1113 | 1112 | 0 | --- |
> | 15 | show_compact_db | acid_3 | 2021 | MAJOR |
> succeeded | 1234 | 1234 | 123456 | 123480 |
> --- | None | --- | --- | --- |
> pool3 | 1113 | 1112 | 0 | --- |
> | 16 | show_compact_db | acid_3 | 2022 | MAJOR |
> working | 1234 | 1234 | 123456 | 123480 |
> --- | None | --- | --- | --- |
> pool1 | 1113 | 1112 | 0 | --- |
> | 17 | show_compact_db | acid_3 | 2022 | MINOR |
> working | 1234 | 1234 | 123456 | 123480 |
> --- | None | --- | --- | --- |
> pool1 | 1113 | 1112 | 0 | --- |
> | 18 | show_compact_db | acid_3 | 2022 | MAJOR |
> succeeded | 1234 | 1234 | 123456 | 123480 |
> --- | None | --- | --- | --- |
> pool1 | 1113 | 1112 | 0 | --- |
> | 19 | show_compact_db | acid_3 | 2022 | MAJOR |
> ready for cleaning | 1234 | 1234 | 123456 | 123480 |
> --- | None | --- | --- | --- |
> pool1 | 1113 | 1112 | 0 | --- |
> +---------------+------------------+----------+-----------+------------+---------------------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
> {code}
> * Run the SHOW COMPACTIONS command with filtering on partition. The result
> is:
> {code:java}
> 0: jdbc:hive2://localhost:10000> show compactions show_compact_db.acid_3
> partition(year="2020");
> +---------------+---------+----------+-----------+-------+--------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
> | compactionid | dbname | tabname | partname | type | state |
> workerhost | workerid | enqueuetime | starttime | duration | hadoopjobid
> | errormessage | initiatorhost | initiatorid | poolname | txnid |
> nexttxnid | committime | hightestwriteid |
> +---------------+---------+----------+-----------+-------+--------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
> +---------------+---------+----------+-----------+-------+--------+-------------+-----------+--------------+------------+-----------+--------------+---------------+----------------+--------------+-----------+--------+------------+-------------+------------------+
>
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
