[ https://issues.apache.org/jira/browse/IMPALA-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17838202#comment-17838202 ]
Quanlong Huang commented on IMPALA-13009: ----------------------------------------- Note that I disabled event processing in the above runs. I found it's more easy to reproduce the issue with event processing turned on, and using the script without the hdfs command: {code:bash} impala-shell.sh -B --quiet -q "create table if not exists my_part(i int) partitioned by (p int); insert into my_part partition(p) values (1,1),(2,2),(3,3)" i=1 while true; do beeline -u "jdbc:hive2://localhost:11050" -e "alter table my_part drop partition(p>0)" impala-shell.sh -B --quiet --ignore_query_failure -q "alter table my_part recover partitions; invalidate metadata my_part; show partitions my_part" i=$((i+3)) impala-shell.sh -B --quiet -q "insert into my_part partition(p) values ($i,$i),($((i+1)),$((i+1))),($((i+2)),$((i+2)))" done {code} Let this run several runs and then restart impalads using {code:bash} bin/start-impala-cluster.py -r{code} Or manually kill one impalad and launch it using commands like {code} source bin/set-classpath.sh /home/quanlong/workspace/Impala/be/build/latest/service/impalad -allow_tuple_caching=true -jni_frontend_class=org/apache/impala/service/JniFrontend -disconnected_session_timeout 21600 -kudu_client_rpc_timeout_ms 0 -kudu_master_hosts localhost -mem_limit=7766732526 -logbufsecs=5 -v=1 -max_log_files=10 -log_rotation_match_pid=true -log_filename=impalad_node2 -log_dir=/home/quanlong/workspace/Impala/logs/cluster -beeswax_port=21002 -hs2_port=21052 -hs2_http_port=28002 -krpc_port=27002 -state_store_subscriber_port=23002 -webserver_port=25002 -num_abfs_io_threads=1 -num_adls_io_threads=1 -num_cos_io_threads=1 -num_gcs_io_threads=1 -num_obs_io_threads=1 -num_oss_io_threads=1 -num_ozone_io_threads=1 -num_s3_io_threads=1 -num_s3_file_oper_io_threads=1 -num_sfs_io_threads=1 -geospatial_library=HIVE_ESRI{code} The log should show the IllegalStateException. > Possible leak of partition updates when the table has failed DDL and > recovered by INVALIDATE METADATA > ----------------------------------------------------------------------------------------------------- > > Key: IMPALA-13009 > URL: https://issues.apache.org/jira/browse/IMPALA-13009 > Project: IMPALA > Issue Type: Bug > Components: Catalog > Reporter: Quanlong Huang > Assignee: Quanlong Huang > Priority: Critical > > Catalogd might not send partition deletions to the catalog topic in the > following scenario: > * Partitions of a table are dropped externally outside Impala. > * Table dir is also removed on HDFS. > * ALTER TABLE RECOVER PARTITIONS failed by FileNotFoundException on the table > dir. > * A subsequent INVALIDATE METADATA on the same table succeeds to invalidate > the table. > After the INVALIDATE finishes, catalogd might not send deletions of the > dropped partitions to the catalog topic. Then the catalog topic only have the > updates of those partitions, no deletions. > This will be detected when a coordinator restarts: > {noformat} > E0417 16:41:22.317298 20746 ImpaladCatalog.java:264] Error adding catalog > object: Received stale partition in a statestore update: > THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL, > type:TColumnType(types:[TTypeNode(type:SCALAR, > scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true, > int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])], > location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138, > file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 > 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 > 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 > 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 > 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 > 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE, > stats:TTableStats(num_rows:-1), is_marked_cached:false, > hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4, > numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4, > has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part, > partition_name:p=106, > hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1, > collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT, > blockSize:0)) > Java exception follows: > java.lang.IllegalStateException: Received stale partition in a statestore > update: > THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL, > type:TColumnType(types:[TTypeNode(type:SCALAR, > scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true, > int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])], > location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138, > file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 > 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 > 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 > 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 > 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 > 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE, > stats:TTableStats(num_rows:-1), is_marked_cached:false, > hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4, > numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4, > has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part, > partition_name:p=106, > hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1, > collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT, > blockSize:0)) > at > com.google.common.base.Preconditions.checkState(Preconditions.java:512) > at > org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:523) > at > org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:334) > at > org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:262) > at > org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:120) > at > org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:565) > at > org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:196) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org