[
https://issues.apache.org/jira/browse/IMPALA-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17838202#comment-17838202
]
Quanlong Huang commented on IMPALA-13009:
-----------------------------------------
Note that I disabled event processing in the above runs. I found it's more easy
to reproduce the issue with event processing turned on, and using the script
without the hdfs command:
{code:bash}
impala-shell.sh -B --quiet -q "create table if not exists my_part(i int)
partitioned by (p int); insert into my_part partition(p) values
(1,1),(2,2),(3,3)"
i=1
while true; do
beeline -u "jdbc:hive2://localhost:11050" -e "alter table my_part drop
partition(p>0)"
impala-shell.sh -B --quiet --ignore_query_failure -q "alter table my_part
recover partitions; invalidate metadata my_part; show partitions my_part"
i=$((i+3))
impala-shell.sh -B --quiet -q "insert into my_part partition(p) values
($i,$i),($((i+1)),$((i+1))),($((i+2)),$((i+2)))"
done {code}
Let this run several runs and then restart impalads using
{code:bash}
bin/start-impala-cluster.py -r{code}
Or manually kill one impalad and launch it using commands like
{code}
source bin/set-classpath.sh
/home/quanlong/workspace/Impala/be/build/latest/service/impalad
-allow_tuple_caching=true
-jni_frontend_class=org/apache/impala/service/JniFrontend
-disconnected_session_timeout 21600 -kudu_client_rpc_timeout_ms 0
-kudu_master_hosts localhost -mem_limit=7766732526 -logbufsecs=5 -v=1
-max_log_files=10 -log_rotation_match_pid=true -log_filename=impalad_node2
-log_dir=/home/quanlong/workspace/Impala/logs/cluster -beeswax_port=21002
-hs2_port=21052 -hs2_http_port=28002 -krpc_port=27002
-state_store_subscriber_port=23002 -webserver_port=25002 -num_abfs_io_threads=1
-num_adls_io_threads=1 -num_cos_io_threads=1 -num_gcs_io_threads=1
-num_obs_io_threads=1 -num_oss_io_threads=1 -num_ozone_io_threads=1
-num_s3_io_threads=1 -num_s3_file_oper_io_threads=1 -num_sfs_io_threads=1
-geospatial_library=HIVE_ESRI{code}
The log should show the IllegalStateException.
> Possible leak of partition updates when the table has failed DDL and
> recovered by INVALIDATE METADATA
> -----------------------------------------------------------------------------------------------------
>
> Key: IMPALA-13009
> URL: https://issues.apache.org/jira/browse/IMPALA-13009
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> Catalogd might not send partition deletions to the catalog topic in the
> following scenario:
> * Partitions of a table are dropped externally outside Impala.
> * Table dir is also removed on HDFS.
> * ALTER TABLE RECOVER PARTITIONS failed by FileNotFoundException on the table
> dir.
> * A subsequent INVALIDATE METADATA on the same table succeeds to invalidate
> the table.
> After the INVALIDATE finishes, catalogd might not send deletions of the
> dropped partitions to the catalog topic. Then the catalog topic only have the
> updates of those partitions, no deletions.
> This will be detected when a coordinator restarts:
> {noformat}
> E0417 16:41:22.317298 20746 ImpaladCatalog.java:264] Error adding catalog
> object: Received stale partition in a statestore update:
> THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL,
> type:TColumnType(types:[TTypeNode(type:SCALAR,
> scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true,
> int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])],
> location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138,
> file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00
> 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01
> 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00
> 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30
> 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74
> 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE,
> stats:TTableStats(num_rows:-1), is_marked_cached:false,
> hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4,
> numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4,
> has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part,
> partition_name:p=106,
> hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1,
> collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT,
> blockSize:0))
> Java exception follows:
> java.lang.IllegalStateException: Received stale partition in a statestore
> update:
> THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL,
> type:TColumnType(types:[TTypeNode(type:SCALAR,
> scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true,
> int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])],
> location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138,
> file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00
> 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01
> 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00
> 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30
> 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74
> 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE,
> stats:TTableStats(num_rows:-1), is_marked_cached:false,
> hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4,
> numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4,
> has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part,
> partition_name:p=106,
> hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1,
> collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT,
> blockSize:0))
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:512)
> at
> org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:523)
> at
> org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:334)
> at
> org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:262)
> at
> org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:120)
> at
> org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:565)
> at
> org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:196)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
