[
https://issues.apache.org/jira/browse/KUDU-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liu closed KUDU-3370.
---------------------
Resolution: Not A Bug
> drop table failed
> -----------------
>
> Key: KUDU-3370
> URL: https://issues.apache.org/jira/browse/KUDU-3370
> Project: Kudu
> Issue Type: Bug
> Components: master, tserver
> Affects Versions: 1.15.0
> Environment: hdp :3.2.3
> hive :3.1.1
> kudu :1.15.0
> trino:369-SNAPSHOT
> Reporter: liu
> Priority: Critical
> Fix For: NA
>
> Attachments: image-2022-05-24-00-11-43-309.png
>
>
> At present, according to the patch provided by hive-16993, hive 3.1.1 is
> recompiled. Now it can successfully synchronize with hive metadata and create
> tables. However, when deleting tables, the log shows that the tables have
> been successfully deleted, but the tables in kudu still exist and cannot be
> deleted。
> {code:java}
> USE
> trino:test2> CREATE TABLE users (
> -> userid int WITH (primary_key = true),
> -> firstname varchar,
> -> lastname varchar
> -> ) WITH (
> -> partition_by_hash_columns = ARRAY['userid'],
> -> partition_by_hash_buckets = 2
> -> );
> CREATE TABLE
> trino:test2> show tables;
> Table
> -------
> users
> (1 row)
> trino:test2> drop table users;
> DROP TABLE
> trino:test2> show tables;
> Table
> -------
> users
> (1 row)Query 20220523_152932_00147_ytvpx, FINISHED, 3 nodes
> Splits: 20 total, 20 done (100.00%)
> 0.23 [1 rows, 20B] [4 rows/s, 86B/s]
> trino:test2> drop table users;
> Query 20220523_152946_00148_ytvpx failed: failed to drop Hive Metastore
> table: TException - service has thrown:
> NoSuchObjectException(message=hive.test2.users table not found) {code}
> The metadata of hive has been successfully deleted when I delete it for the
> first time. When I delete it for the second time, it is reported that the
> metadata does not exist because it has been deleted, but the table still
> exists in kudu. Use the command:
> {code:java}
> kudu table delete gpnode1 test2.users{code}
> error message：
> {code:java}
> # kudu table delete gpnode1 test2.users
> # Not found: failed to drop Hive Metastore table: TException - service has
> thrown: NoSuchObjectException(message=hive.test2.users table not found) {code}
> log：
> {code:java}
> W0523 23:21:52.950664 11595 master_path_handlers.cc:837] Unable to fetch
> master addresses: OK
> W0523 23:22:31.867633 11595 master_path_handlers.cc:837] Unable to fetch
> master addresses: OK
> I0523 23:28:41.818248 5917 catalog_manager.cc:1681] Servicing CreateTable
> request from {username='presto'} at 192.168.2.182:36402:
> name: "test2.users"
> schema {
> columns {
> name: "userid"
> type: INT32
> is_key: true
> is_nullable: false
> cfile_block_size: 0
> }
> columns {
> name: "firstname"
> type: STRING
> is_key: false
> is_nullable: false
> cfile_block_size: 0
> }
> columns {
> name: "lastname"
> type: STRING
> is_key: false
> is_nullable: false
> cfile_block_size: 0
> }
> }
> partition_schema {
> hash_bucket_schemas {
> columns {
> name: "userid"
> }
> num_buckets: 2
> seed: 0
> }
> }
> I0523 23:28:41.877279 5917 catalog_manager.cc:1963] created HMS catalog
> entry for table [id=3a50aa7d5aa6416f8ab5587ed16bc85a]
> I0523 23:28:42.091735 5907 maintenance_manager.cc:382] P
> a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
> FlushMRSOp(00000000000000000000000000000000): perf score=0.302975
> I0523 23:28:42.109527 5906 maintenance_manager.cc:603] P
> a01491afc9f24dc3adef5d9cbc4ff405:
> FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.018s
> user 0.003s sys 0.003s Metrics:
> {"bytes_written":5660,"cfile_init":1,"dirs.queue_time_us":90,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":667,"drs_written":1,"fdatasync":13,"fdatasync_us":11633,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4,"lbm_write_time_us":133,"lbm_writes_lt_1ms":27,"rows_written":3,"thread_start_us":59,"threads_started":1}
> I0523 23:28:42.109880 5907 maintenance_manager.cc:382] P
> a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
> CompactRowSetsOp(00000000000000000000000000000000): perf score=0.212344
> I0523 23:28:42.133885 5906 maintenance_manager.cc:603] P
> a01491afc9f24dc3adef5d9cbc4ff405:
> CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real
> 0.024s user 0.008s sys 0.004s Metrics:
> {"bytes_written":213198,"cfile_cache_hit":13,"cfile_cache_hit_bytes":206138,"cfile_cache_miss":35,"cfile_cache_miss_bytes":225520,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":502,"dirs.run_cpu_time_us":810,"dirs.run_wall_time_us":950,"drs_written":1,"fdatasync":15,"fdatasync_us":12534,"lbm_read_time_us":662,"lbm_reads_lt_1ms":75,"lbm_write_time_us":530,"lbm_writes_lt_1ms":32,"num_input_rowsets":2,"rows_written":209,"spinlock_wait_cycles":2944,"thread_start_us":397,"threads_started":4}
> I0523 23:28:44.408016 5917 catalog_manager.cc:4733] T
> 273e6e3223f14ea690621e40c53c8c0e P 18c9a3f5aa314a2b80dfa970003e4350 reported
> cstate change: term changed from 0 to 1, leader changed from <none> to
> 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211). New cstate: current_term: 1
> leader_uuid: "18c9a3f5aa314a2b80dfa970003e4350" committed_config {
> opid_index: -1 OBSOLETE_local: false peers { permanent_uuid:
> "524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host:
> "192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } }
> peers { permanent_uuid: "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER
> last_known_addr { host: "192.168.2.211" port: 7050 } health_report {
> overall_health: HEALTHY } } peers { permanent_uuid:
> "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER last_known_addr { host:
> "192.168.2.182" port: 7050 } health_report { overall_health: UNKNOWN } } }
> I0523 23:28:59.145552 5917 catalog_manager.cc:4733] T
> e0d2911469d147609abcdcebffacc7f1 P 18c9a3f5aa314a2b80dfa970003e4350 reported
> cstate change: term changed from 0 to 1, leader changed from <none> to
> 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211). New cstate: current_term: 1
> leader_uuid: "18c9a3f5aa314a2b80dfa970003e4350" committed_config {
> opid_index: -1 OBSOLETE_local: false peers { permanent_uuid:
> "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host:
> "192.168.2.211" port: 7050 } health_report { overall_health: HEALTHY } }
> peers { permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER
> last_known_addr { host: "192.168.2.182" port: 7050 } health_report {
> overall_health: UNKNOWN } } peers { permanent_uuid:
> "524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host:
> "192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } }
> I0523 23:29:28.640434 5917 catalog_manager.cc:2253] Servicing DeleteTable
> request from {username='presto'} at 192.168.2.182:36402:
> table { table_name: "test2.users" }
> I0523 23:29:28.706779 5917 catalog_manager.cc:2293] deleted HMS catalog
> entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]
> I0523 23:29:33.955879 5917 catalog_manager.cc:4733] T
> e0d2911469d147609abcdcebffacc7f1 P 9ad2b92eba814e78895e645f74618bd2 reported
> cstate change: term changed from 1 to 2, leader changed from
> 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211) to
> 9ad2b92eba814e78895e645f74618bd2 (192.168.2.182). New cstate: current_term: 2
> leader_uuid: "9ad2b92eba814e78895e645f74618bd2" committed_config {
> opid_index: -1 OBSOLETE_local: false peers { permanent_uuid:
> "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host:
> "192.168.2.211" port: 7050 } health_report { overall_health: UNKNOWN } }
> peers { permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER
> last_known_addr { host: "192.168.2.182" port: 7050 } health_report {
> overall_health: HEALTHY } } peers { permanent_uuid:
> "524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host:
> "192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } }
> I0523 23:29:46.440908 5917 catalog_manager.cc:2253] Servicing DeleteTable
> request from {username='presto'} at 192.168.2.182:36402:
> table { table_name: "test2.users" }
> W0523 23:29:46.446835 5917 catalog_manager.cc:2296] failed to delete HMS
> catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]:
> Not found: failed to drop Hive Metastore table: TException - service has
> thrown: NoSuchObjectException(message=hive.test2.users table not found)
> I0523 23:30:44.479030 5907 maintenance_manager.cc:382] P
> a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
> FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033352
> I0523 23:30:44.493083 5906 maintenance_manager.cc:603] P
> a01491afc9f24dc3adef5d9cbc4ff405:
> FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing:
> real 0.014s user 0.001s sys 0.000s Metrics:
> {"bytes_written":476,"delete_count":0,"fdatasync":3,"fdatasync_us":12119,"lbm_write_time_us":23,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":21,"threads_started":1,"update_count":3,"wal-append.queue_time_us":63}
> I0523 23:30:44.493170 5907 maintenance_manager.cc:382] P
> a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
> MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
> I0523 23:30:44.498075 5906 maintenance_manager.cc:603] P
> a01491afc9f24dc3adef5d9cbc4ff405:
> MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing:
> real 0.005s user 0.001s sys 0.000s Metrics:
> {"cfile_cache_hit":1,"cfile_cache_hit_bytes":29,"cfile_cache_miss":3,"cfile_cache_miss_bytes":1283,"cfile_init":1,"delete_count":146,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"fdatasync":3,"fdatasync_us":3378,"lbm_read_time_us":5,"lbm_reads_lt_1ms":7,"lbm_write_time_us":11,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":16,"threads_started":1,"update_count":3}
> I0524 00:00:48.336750 5917 catalog_manager.cc:2253] Servicing DeleteTable
> request from {username='root'} at 192.168.2.182:42712:
> table { table_name: "test2.users" } modify_external_catalogs: true
> W0524 00:00:48.349134 5917 catalog_manager.cc:2296] failed to delete HMS
> catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]:
> Not found: failed to drop Hive Metastore table: TException - service has
> thrown: NoSuchObjectException(message=hive.test2.users table not found)
> {code}
>
>
> !image-2022-05-24-00-11-43-309.png!
>
> kudu cluster ksck gpmaster,gpnode1,gpnode2
> {code:java}
> [kudu@gpnode1 kudu]$ kudu cluster ksck gpmaster,gpnode1,gpnode2
> Master Summary
> UUID | Address | Status
> ----------------------------------+----------+---------
> 2bc3b2318ca640a78a99fcbe4d058a9f | gpmaster | HEALTHY
> a01491afc9f24dc3adef5d9cbc4ff405 | gpnode1 | HEALTHY
> ccbd77b62e784359b71c07d16a909e59 | gpnode2 | HEALTHYUnusual flags for
> Master:
> Flag | Value | Tags | Master
> ------------------+-------+--------+-------------------------
> use_hybrid_clock | false | hidden | all 3 server(s) checkedFlags of checked
> categories for Master:
> Flag | Value
> | Master
> ---------------------+-------------------------------------------------------------+-------------------------
> builtin_ntp_servers |
> 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s)
> checked
> time_source | system
> |
> all 3 server(s) checkedTablet Server Summary
> UUID | Address | Status | Location |
> Tablet Leaders |
> Active Scanners
> ----------------------------------+--------------------+---------+----------+----------------+-----------------
> 18c9a3f5aa314a2b80dfa970003e4350 | 192.168.2.211:7050 | HEALTHY | <none> |
> 0 | 0
> 524c9034d0b540cab9161d44a5dab7ef | 192.168.2.184:7050 | HEALTHY | <none> |
> 14 | 0
> 7f33079cf18941eda06a0fcc06c3afd3 | 192.168.2.183:7050 | HEALTHY | <none> |
> 11 | 0
> 9ad2b92eba814e78895e645f74618bd2 | 192.168.2.182:7050 | HEALTHY | <none> |
> 13 | 0
> Tablet Server Location Summary
> Location | Count
> ----------+---------
> <none> | 4Unusual flags for Tablet Server:
> Flag | Value | Tags | Tablet Server
> ------------------+-------+--------+-------------------------
> use_hybrid_clock | false | hidden | all 4 server(s) checkedFlags of checked
> categories for Tablet Server:
> Flag | Value
> | Tablet Server
> ---------------------+-------------------------------------------------------------+-------------------------
> builtin_ntp_servers |
> 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s)
> checked
> time_source | system
> |
> all 4 server(s) checkedVersion Summary
> Version | Servers
> ---------+-------------------------
> 1.15.0 | all 7 server(s) checkedTablet Summary
> Summary by table
> Name | RF | Status | Total Tablets | Healthy | Recovering |
> Under-replicated | Unavailable
> ---------------+----+---------+---------------+---------+------------+------------------+-------------
> $schemas | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users1 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users10 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users11 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users12 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users13 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users2 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users3 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users4 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users5 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users6 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users7 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users8 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users9 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test2.users | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test2.users1 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test2.users2 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test2.users3 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0Tablet Replica Count Summary
> Statistic | Replica Count
> ----------------+---------------
> Minimum | 10
> First Quartile | 34
> Median | 35
> Third Quartile | 35
> Maximum | 35
> Tablet Replica Count Outliers
> Type | UUID | Host | Replica Count
> -------+----------------------------------+--------------------+---------------
> Small | 18c9a3f5aa314a2b80dfa970003e4350 | 192.168.2.211:7050 | 10
> Total Count Summary
> | Total Count
> ----------------+-------------
> Masters | 3
> Tablet Servers | 4
> Tables | 19
> Tablets | 38
> Replicas | 114
> ==================
> Warnings:
> ==================
> Some masters have unsafe, experimental, or hidden flags set
> Some tablet servers have unsafe, experimental, or hidden flags set
> OK
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
