Quanlong Huang created IMPALA-13009:
---------------------------------------
Summary: Possible leak of partition updates when the table has
failed DDL and recovered by INVALIDATE METADATA
Key: IMPALA-13009
URL: https://issues.apache.org/jira/browse/IMPALA-13009
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Quanlong Huang
Assignee: Quanlong Huang
Catalogd might not send partition deletions to the catalog topic in the
following scenario:
* Partitions of a table are dropped externally outside Impala.
* Table dir is also removed on HDFS.
* ALTER TABLE RECOVER PARTITIONS failed by FileNotFoundException on the table
dir.
* A subsequent INVALIDATE METADATA on the same table succeeds to invalidated
the table.
After the INVALIDATE finishes, catalogd might not send deletions of the dropped
partitions to the catalog topic. Then the catalog topic only have the updates
of those partitions, no deletions.
This will be detected when a coordinator restarts:
{noformat}
E0417 16:41:22.317298 20746 ImpaladCatalog.java:264] Error adding catalog
object: Received stale partition in a statestore update:
THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL,
type:TColumnType(types:[TTypeNode(type:SCALAR,
scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true,
int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])],
location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138,
file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 1C
00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 00
00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 34
34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 30
30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 78
74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE,
stats:TTableStats(num_rows:-1), is_marked_cached:false,
hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4,
numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4,
has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part,
partition_name:p=106,
hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1,
collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT,
blockSize:0))
Java exception follows:
java.lang.IllegalStateException: Received stale partition in a statestore
update:
THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL,
type:TColumnType(types:[TTypeNode(type:SCALAR,
scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true,
int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])],
location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138,
file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 1C
00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 00
00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 34
34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 30
30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 78
74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE,
stats:TTableStats(num_rows:-1), is_marked_cached:false,
hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4,
numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4,
has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part,
partition_name:p=106,
hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1,
collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT,
blockSize:0))
at
com.google.common.base.Preconditions.checkState(Preconditions.java:512)
at
org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:523)
at
org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:334)
at
org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:262)
at
org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:120)
at
org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:565)
at
org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:196)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
