[
https://issues.apache.org/jira/browse/IMPALA-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-13983:
------------------------------------
Description:
I'm reproducing the issue using the master branch at commit
b19331b3d336f680f67c15b4674e2523288d7e59 with an additional sleep before
sending the alter_table HMS RPC in
[CatalogOpExecutor#applyAlterTable()|https://github.com/apache/impala/blob/b19331b3d336f680f67c15b4674e2523288d7e59/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L6731]:
{code:java}
diff --git a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
index 4461c03f5..5f5dffb40 100644
--- a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
+++ b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
@@ -6728,6 +6728,7 @@ public class CatalogOpExecutor {
MetastoreShim.alterTableWithTransaction(msClient.getHiveClient(),
msTbl, tblTxn);
} else {
try {
+ try {Thread.sleep(20000);} catch (Exception e) {}
msClient.getHiveClient().alter_table(
msTbl.getDbName(), msTbl.getTableName(), msTbl);
} catch (TException e) {{code}
The sleep is added to make the concurrent issue easier to happen.
Steps to reproduce the issue:
Start Impala with HMS EventProcessor disabled (to be simple). Create a table.
{noformat}
bin/start-impala-cluster.py --catalogd_args=--hms_event_polling_interval_s=0
impala> create table tbl (i int);{noformat}
Run the first AlterTable statement in background:
{code:bash}
impala-shell.sh -q "alter table tbl set tblproperties('k1'='100')" &{code}
Then run another session to submit the Invalidate+AlterTable on the same table,
but setting a new tblproperty:
{code:bash}
impala-shell.sh -q "invalidate metadata tbl; alter table tbl set
tblproperties('k2'='200')"{code}
After both of them finish, describe the table to check the tblproperties.
k1=100 is missing there.
{noformat}
Query: describe formatted tbl
+------------------------------+------------------------------------------------------------+----------------------+
| name | type
| comment |
+------------------------------+------------------------------------------------------------+----------------------+
| # col_name | data_type
| comment |
| | NULL
| NULL |
| i | int
| NULL |
| | NULL
| NULL |
| # Detailed Table Information | NULL
| NULL |
| Database: | default
| NULL |
| OwnerType: | USER
| NULL |
| Owner: | quanlong
| NULL |
| CreateTime: | Tue Apr 22 18:37:00 CST 2025
| NULL |
| LastAccessTime: | UNKNOWN
| NULL |
| Retention: | 0
| NULL |
| Location: | hdfs://localhost:20500/test-warehouse/tbl
| NULL |
| Erasure Coding Policy: | NONE
| NULL |
| Table Type: | EXTERNAL_TABLE
| NULL |
| Table Parameters: | NULL
| NULL |
| | EXTERNAL
| TRUE |
| | OBJCAPABILITIES
| EXTREAD,EXTWRITE |
| | TRANSLATED_TO_EXTERNAL
| TRUE |
| | external.table.purge
| TRUE |
| | k2
| 200 |
| | transient_lastDdlTime
| 1745318281 |
| | NULL
| NULL |
| # Storage Information | NULL
| NULL |
| SerDe Library: |
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL
|
| InputFormat: | org.apache.hadoop.mapred.TextInputFormat
| NULL |
| OutputFormat: |
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL
|
| Compressed: | No
| NULL |
| Num Buckets: | 0
| NULL |
| Bucket Columns: | []
| NULL |
| Sort Columns: | []
| NULL |
| | NULL
| NULL |
| # Constraints | NULL
| NULL |
+------------------------------+------------------------------------------------------------+----------------------+{noformat}
CC [~csringhofer] , [~MikaelSmith] , [~rizaon]
was:
I'm reproducing the issue using the master branch at commit
b19331b3d336f680f67c15b4674e2523288d7e59 with an additional sleep before
sending the alter_table HMS RPC in
[CatalogOpExecutor#applyAlterTable()|https://github.com/apache/impala/blob/b19331b3d336f680f67c15b4674e2523288d7e59/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L6731]:
{code:java}
diff --git a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
index 4461c03f5..5f5dffb40 100644
--- a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
+++ b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
@@ -6728,6 +6728,7 @@ public class CatalogOpExecutor {
MetastoreShim.alterTableWithTransaction(msClient.getHiveClient(),
msTbl, tblTxn);
} else {
try {
+ try {Thread.sleep(20000);} catch (Exception e) {}
msClient.getHiveClient().alter_table(
msTbl.getDbName(), msTbl.getTableName(), msTbl);
} catch (TException e) {{code}
The sleep is added to make the concurrent issue easier to happen.
Steps to reproduce the issue:
{code:sql}
create table tbl (i int);{code}
Run the first AlterTable statement in background:
{code:bash}
impala-shell.sh -q "alter table tbl set tblproperties('k1'='100')" &{code}
Then run another session to submit the Invalidate+AlterTable on the same table,
but setting a new tblproperty:
{code:bash}
impala-shell.sh -q "invalidate metadata tbl; alter table tbl set
tblproperties('k2'='200')"{code}
After both of them finish, describe the table to check the tblproperties.
k1=100 is missing there.
{noformat}
Query: describe formatted tbl
+------------------------------+------------------------------------------------------------+----------------------+
| name | type
| comment |
+------------------------------+------------------------------------------------------------+----------------------+
| # col_name | data_type
| comment |
| | NULL
| NULL |
| i | int
| NULL |
| | NULL
| NULL |
| # Detailed Table Information | NULL
| NULL |
| Database: | default
| NULL |
| OwnerType: | USER
| NULL |
| Owner: | quanlong
| NULL |
| CreateTime: | Tue Apr 22 18:37:00 CST 2025
| NULL |
| LastAccessTime: | UNKNOWN
| NULL |
| Retention: | 0
| NULL |
| Location: | hdfs://localhost:20500/test-warehouse/tbl
| NULL |
| Erasure Coding Policy: | NONE
| NULL |
| Table Type: | EXTERNAL_TABLE
| NULL |
| Table Parameters: | NULL
| NULL |
| | EXTERNAL
| TRUE |
| | OBJCAPABILITIES
| EXTREAD,EXTWRITE |
| | TRANSLATED_TO_EXTERNAL
| TRUE |
| | external.table.purge
| TRUE |
| | k2
| 200 |
| | transient_lastDdlTime
| 1745318281 |
| | NULL
| NULL |
| # Storage Information | NULL
| NULL |
| SerDe Library: |
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL
|
| InputFormat: | org.apache.hadoop.mapred.TextInputFormat
| NULL |
| OutputFormat: |
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL
|
| Compressed: | No
| NULL |
| Num Buckets: | 0
| NULL |
| Bucket Columns: | []
| NULL |
| Sort Columns: | []
| NULL |
| | NULL
| NULL |
| # Constraints | NULL
| NULL |
+------------------------------+------------------------------------------------------------+----------------------+{noformat}
CC [~csringhofer] , [~MikaelSmith] , [~rizaon]
> Invalidate+AlterTable could overwrite the result of a concurrent AlterTable
> ---------------------------------------------------------------------------
>
> Key: IMPALA-13983
> URL: https://issues.apache.org/jira/browse/IMPALA-13983
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Priority: Major
>
> I'm reproducing the issue using the master branch at commit
> b19331b3d336f680f67c15b4674e2523288d7e59 with an additional sleep before
> sending the alter_table HMS RPC in
> [CatalogOpExecutor#applyAlterTable()|https://github.com/apache/impala/blob/b19331b3d336f680f67c15b4674e2523288d7e59/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L6731]:
> {code:java}
> diff --git
> a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
> b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
> index 4461c03f5..5f5dffb40 100644
> --- a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
> +++ b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
> @@ -6728,6 +6728,7 @@ public class CatalogOpExecutor {
> MetastoreShim.alterTableWithTransaction(msClient.getHiveClient(),
> msTbl, tblTxn);
> } else {
> try {
> + try {Thread.sleep(20000);} catch (Exception e) {}
> msClient.getHiveClient().alter_table(
> msTbl.getDbName(), msTbl.getTableName(), msTbl);
> } catch (TException e) {{code}
> The sleep is added to make the concurrent issue easier to happen.
> Steps to reproduce the issue:
> Start Impala with HMS EventProcessor disabled (to be simple). Create a table.
> {noformat}
> bin/start-impala-cluster.py --catalogd_args=--hms_event_polling_interval_s=0
> impala> create table tbl (i int);{noformat}
> Run the first AlterTable statement in background:
> {code:bash}
> impala-shell.sh -q "alter table tbl set tblproperties('k1'='100')" &{code}
> Then run another session to submit the Invalidate+AlterTable on the same
> table, but setting a new tblproperty:
> {code:bash}
> impala-shell.sh -q "invalidate metadata tbl; alter table tbl set
> tblproperties('k2'='200')"{code}
> After both of them finish, describe the table to check the tblproperties.
> k1=100 is missing there.
> {noformat}
> Query: describe formatted tbl
> +------------------------------+------------------------------------------------------------+----------------------+
> | name | type
> | comment |
> +------------------------------+------------------------------------------------------------+----------------------+
> | # col_name | data_type
> | comment |
> | | NULL
> | NULL |
> | i | int
> | NULL |
> | | NULL
> | NULL |
> | # Detailed Table Information | NULL
> | NULL |
> | Database: | default
> | NULL |
> | OwnerType: | USER
> | NULL |
> | Owner: | quanlong
> | NULL |
> | CreateTime: | Tue Apr 22 18:37:00 CST 2025
> | NULL |
> | LastAccessTime: | UNKNOWN
> | NULL |
> | Retention: | 0
> | NULL |
> | Location: | hdfs://localhost:20500/test-warehouse/tbl
> | NULL |
> | Erasure Coding Policy: | NONE
> | NULL |
> | Table Type: | EXTERNAL_TABLE
> | NULL |
> | Table Parameters: | NULL
> | NULL |
> | | EXTERNAL
> | TRUE |
> | | OBJCAPABILITIES
> | EXTREAD,EXTWRITE |
> | | TRANSLATED_TO_EXTERNAL
> | TRUE |
> | | external.table.purge
> | TRUE |
> | | k2
> | 200 |
> | | transient_lastDdlTime
> | 1745318281 |
> | | NULL
> | NULL |
> | # Storage Information | NULL
> | NULL |
> | SerDe Library: |
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL
> |
> | InputFormat: | org.apache.hadoop.mapred.TextInputFormat
> | NULL |
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL
> |
> | Compressed: | No
> | NULL |
> | Num Buckets: | 0
> | NULL |
> | Bucket Columns: | []
> | NULL |
> | Sort Columns: | []
> | NULL |
> | | NULL
> | NULL |
> | # Constraints | NULL
> | NULL |
> +------------------------------+------------------------------------------------------------+----------------------+{noformat}
> CC [~csringhofer] , [~MikaelSmith] , [~rizaon]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
