[
https://issues.apache.org/jira/browse/IMPALA-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-13009:
------------------------------------
Description:
Catalogd might not send partition deletions to the catalog topic in the
following scenario:
* Partitions of a table are dropped externally outside Impala.
* Table dir is also removed on HDFS.
* ALTER TABLE RECOVER PARTITIONS failed by FileNotFoundException on the table
dir.
* A subsequent INVALIDATE METADATA on the same table succeeds to invalidate the
table.
After the INVALIDATE finishes, catalogd might not send deletions of the dropped
partitions to the catalog topic. Then the catalog topic only have the updates
of those partitions, no deletions.
This will be detected when a coordinator restarts:
{noformat}
E0417 16:41:22.317298 20746 ImpaladCatalog.java:264] Error adding catalog
object: Received stale partition in a statestore update:
THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL,
type:TColumnType(types:[TTypeNode(type:SCALAR,
scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true,
int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])],
location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138,
file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 1C
00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 00
00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 34
34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 30
30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 78
74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE,
stats:TTableStats(num_rows:-1), is_marked_cached:false,
hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4,
numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4,
has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part,
partition_name:p=106,
hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1,
collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT,
blockSize:0))
Java exception follows:
java.lang.IllegalStateException: Received stale partition in a statestore
update:
THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL,
type:TColumnType(types:[TTypeNode(type:SCALAR,
scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true,
int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])],
location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138,
file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 1C
00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 00
00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 34
34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 30
30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 78
74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE,
stats:TTableStats(num_rows:-1), is_marked_cached:false,
hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4,
numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4,
has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part,
partition_name:p=106,
hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1,
collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT,
blockSize:0))
at
com.google.common.base.Preconditions.checkState(Preconditions.java:512)
at
org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:523)
at
org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:334)
at
org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:262)
at
org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:120)
at
org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:565)
at
org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:196)
{noformat}
was:
Catalogd might not send partition deletions to the catalog topic in the
following scenario:
* Partitions of a table are dropped externally outside Impala.
* Table dir is also removed on HDFS.
* ALTER TABLE RECOVER PARTITIONS failed by FileNotFoundException on the table
dir.
* A subsequent INVALIDATE METADATA on the same table succeeds to invalidated
the table.
After the INVALIDATE finishes, catalogd might not send deletions of the dropped
partitions to the catalog topic. Then the catalog topic only have the updates
of those partitions, no deletions.
This will be detected when a coordinator restarts:
{noformat}
E0417 16:41:22.317298 20746 ImpaladCatalog.java:264] Error adding catalog
object: Received stale partition in a statestore update:
THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL,
type:TColumnType(types:[TTypeNode(type:SCALAR,
scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true,
int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])],
location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138,
file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 1C
00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 00
00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 34
34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 30
30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 78
74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE,
stats:TTableStats(num_rows:-1), is_marked_cached:false,
hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4,
numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4,
has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part,
partition_name:p=106,
hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1,
collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT,
blockSize:0))
Java exception follows:
java.lang.IllegalStateException: Received stale partition in a statestore
update:
THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL,
type:TColumnType(types:[TTypeNode(type:SCALAR,
scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true,
int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])],
location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138,
file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 1C
00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 00
00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 34
34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 30
30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 78
74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE,
stats:TTableStats(num_rows:-1), is_marked_cached:false,
hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4,
numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4,
has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part,
partition_name:p=106,
hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1,
collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT,
blockSize:0))
at
com.google.common.base.Preconditions.checkState(Preconditions.java:512)
at
org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:523)
at
org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:334)
at
org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:262)
at
org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:120)
at
org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:565)
at
org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:196)
{noformat}
> Possible leak of partition updates when the table has failed DDL and
> recovered by INVALIDATE METADATA
> -----------------------------------------------------------------------------------------------------
>
> Key: IMPALA-13009
> URL: https://issues.apache.org/jira/browse/IMPALA-13009
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> Catalogd might not send partition deletions to the catalog topic in the
> following scenario:
> * Partitions of a table are dropped externally outside Impala.
> * Table dir is also removed on HDFS.
> * ALTER TABLE RECOVER PARTITIONS failed by FileNotFoundException on the table
> dir.
> * A subsequent INVALIDATE METADATA on the same table succeeds to invalidate
> the table.
> After the INVALIDATE finishes, catalogd might not send deletions of the
> dropped partitions to the catalog topic. Then the catalog topic only have the
> updates of those partitions, no deletions.
> This will be detected when a coordinator restarts:
> {noformat}
> E0417 16:41:22.317298 20746 ImpaladCatalog.java:264] Error adding catalog
> object: Received stale partition in a statestore update:
> THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL,
> type:TColumnType(types:[TTypeNode(type:SCALAR,
> scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true,
> int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])],
> location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138,
> file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00
> 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01
> 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00
> 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30
> 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74
> 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE,
> stats:TTableStats(num_rows:-1), is_marked_cached:false,
> hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4,
> numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4,
> has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part,
> partition_name:p=106,
> hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1,
> collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT,
> blockSize:0))
> Java exception follows:
> java.lang.IllegalStateException: Received stale partition in a statestore
> update:
> THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL,
> type:TColumnType(types:[TTypeNode(type:SCALAR,
> scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true,
> int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])],
> location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138,
> file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00
> 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01
> 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00
> 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30
> 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74
> 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE,
> stats:TTableStats(num_rows:-1), is_marked_cached:false,
> hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4,
> numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4,
> has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part,
> partition_name:p=106,
> hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1,
> collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT,
> blockSize:0))
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:512)
> at
> org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:523)
> at
> org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:334)
> at
> org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:262)
> at
> org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:120)
> at
> org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:565)
> at
> org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:196)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
