[
https://issues.apache.org/jira/browse/KUDU-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liu updated KUDU-3370:
----------------------
Description:
At present, according to the patch provided by hive-16993, hive 3.1.1 is
recompiled. Now it can successfully synchronize with hive metadata and create
tables. However, when deleting tables, the log shows that the tables have been
successfully deleted, but the tables in kudu still exist and cannot be deleted。
{code:java}
USE
trino:test2> CREATE TABLE users (
-> userid int WITH (primary_key = true),
-> firstname varchar,
-> lastname varchar
-> ) WITH (
-> partition_by_hash_columns = ARRAY['userid'],
-> partition_by_hash_buckets = 2
-> );
CREATE TABLE
trino:test2> show tables;
Table
-------
users
(1 row)
trino:test2> drop table users;
DROP TABLE
trino:test2> show tables;
Table
-------
users
(1 row)Query 20220523_152932_00147_ytvpx, FINISHED, 3 nodes
Splits: 20 total, 20 done (100.00%)
0.23 [1 rows, 20B] [4 rows/s, 86B/s]
trino:test2> drop table users;
Query 20220523_152946_00148_ytvpx failed: failed to drop Hive Metastore table:
TException - service has thrown: NoSuchObjectException(message=hive.test2.users
table not found) {code}
The metadata of hive has been successfully deleted when I delete it for the
first time. When I delete it for the second time, it is reported that the
metadata does not exist because it has been deleted, but the table still exists
in kudu. Use the command:
{code:java}
kudu table delete gpnode1 test2.users{code}
error message：
{code:java}
# kudu table delete gpnode1 test2.users
# Not found: failed to drop Hive Metastore table: TException - service has
thrown: NoSuchObjectException(message=hive.test2.users table not found) {code}
log：
{code:java}
W0523 23:21:52.950664 11595 master_path_handlers.cc:837] Unable to fetch master
addresses: OK
W0523 23:22:31.867633 11595 master_path_handlers.cc:837] Unable to fetch master
addresses: OK
I0523 23:28:41.818248 5917 catalog_manager.cc:1681] Servicing CreateTable
request from {username='presto'} at 192.168.2.182:36402:
name: "test2.users"
schema {
columns {
name: "userid"
type: INT32
is_key: true
is_nullable: false
cfile_block_size: 0
}
columns {
name: "firstname"
type: STRING
is_key: false
is_nullable: false
cfile_block_size: 0
}
columns {
name: "lastname"
type: STRING
is_key: false
is_nullable: false
cfile_block_size: 0
}
}
partition_schema {
hash_bucket_schemas {
columns {
name: "userid"
}
num_buckets: 2
seed: 0
}
}
I0523 23:28:41.877279 5917 catalog_manager.cc:1963] created HMS catalog entry
for table [id=3a50aa7d5aa6416f8ab5587ed16bc85a]
I0523 23:28:42.091735 5907 maintenance_manager.cc:382] P
a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
FlushMRSOp(00000000000000000000000000000000): perf score=0.302975
I0523 23:28:42.109527 5906 maintenance_manager.cc:603] P
a01491afc9f24dc3adef5d9cbc4ff405: FlushMRSOp(00000000000000000000000000000000)
complete. Timing: real 0.018s user 0.003s sys 0.003s Metrics:
{"bytes_written":5660,"cfile_init":1,"dirs.queue_time_us":90,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":667,"drs_written":1,"fdatasync":13,"fdatasync_us":11633,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4,"lbm_write_time_us":133,"lbm_writes_lt_1ms":27,"rows_written":3,"thread_start_us":59,"threads_started":1}
I0523 23:28:42.109880 5907 maintenance_manager.cc:382] P
a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
CompactRowSetsOp(00000000000000000000000000000000): perf score=0.212344
I0523 23:28:42.133885 5906 maintenance_manager.cc:603] P
a01491afc9f24dc3adef5d9cbc4ff405:
CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real
0.024s user 0.008s sys 0.004s Metrics:
{"bytes_written":213198,"cfile_cache_hit":13,"cfile_cache_hit_bytes":206138,"cfile_cache_miss":35,"cfile_cache_miss_bytes":225520,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":502,"dirs.run_cpu_time_us":810,"dirs.run_wall_time_us":950,"drs_written":1,"fdatasync":15,"fdatasync_us":12534,"lbm_read_time_us":662,"lbm_reads_lt_1ms":75,"lbm_write_time_us":530,"lbm_writes_lt_1ms":32,"num_input_rowsets":2,"rows_written":209,"spinlock_wait_cycles":2944,"thread_start_us":397,"threads_started":4}
I0523 23:28:44.408016 5917 catalog_manager.cc:4733] T
273e6e3223f14ea690621e40c53c8c0e P 18c9a3f5aa314a2b80dfa970003e4350 reported
cstate change: term changed from 0 to 1, leader changed from <none> to
18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211). New cstate: current_term: 1
leader_uuid: "18c9a3f5aa314a2b80dfa970003e4350" committed_config { opid_index:
-1 OBSOLETE_local: false peers { permanent_uuid:
"524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host:
"192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } peers
{ permanent_uuid: "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER
last_known_addr { host: "192.168.2.211" port: 7050 } health_report {
overall_health: HEALTHY } } peers { permanent_uuid:
"9ad2b92eba814e78895e645f74618bd2" member_type: VOTER last_known_addr { host:
"192.168.2.182" port: 7050 } health_report { overall_health: UNKNOWN } } }
I0523 23:28:59.145552 5917 catalog_manager.cc:4733] T
e0d2911469d147609abcdcebffacc7f1 P 18c9a3f5aa314a2b80dfa970003e4350 reported
cstate change: term changed from 0 to 1, leader changed from <none> to
18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211). New cstate: current_term: 1
leader_uuid: "18c9a3f5aa314a2b80dfa970003e4350" committed_config { opid_index:
-1 OBSOLETE_local: false peers { permanent_uuid:
"18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host:
"192.168.2.211" port: 7050 } health_report { overall_health: HEALTHY } } peers
{ permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER
last_known_addr { host: "192.168.2.182" port: 7050 } health_report {
overall_health: UNKNOWN } } peers { permanent_uuid:
"524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host:
"192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } }
I0523 23:29:28.640434 5917 catalog_manager.cc:2253] Servicing DeleteTable
request from {username='presto'} at 192.168.2.182:36402:
table { table_name: "test2.users" }
I0523 23:29:28.706779 5917 catalog_manager.cc:2293] deleted HMS catalog entry
for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]
I0523 23:29:33.955879 5917 catalog_manager.cc:4733] T
e0d2911469d147609abcdcebffacc7f1 P 9ad2b92eba814e78895e645f74618bd2 reported
cstate change: term changed from 1 to 2, leader changed from
18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211) to
9ad2b92eba814e78895e645f74618bd2 (192.168.2.182). New cstate: current_term: 2
leader_uuid: "9ad2b92eba814e78895e645f74618bd2" committed_config { opid_index:
-1 OBSOLETE_local: false peers { permanent_uuid:
"18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host:
"192.168.2.211" port: 7050 } health_report { overall_health: UNKNOWN } } peers
{ permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER
last_known_addr { host: "192.168.2.182" port: 7050 } health_report {
overall_health: HEALTHY } } peers { permanent_uuid:
"524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host:
"192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } }
I0523 23:29:46.440908 5917 catalog_manager.cc:2253] Servicing DeleteTable
request from {username='presto'} at 192.168.2.182:36402:
table { table_name: "test2.users" }
W0523 23:29:46.446835 5917 catalog_manager.cc:2296] failed to delete HMS
catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]: Not
found: failed to drop Hive Metastore table: TException - service has thrown:
NoSuchObjectException(message=hive.test2.users table not found)
I0523 23:30:44.479030 5907 maintenance_manager.cc:382] P
a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033352
I0523 23:30:44.493083 5906 maintenance_manager.cc:603] P
a01491afc9f24dc3adef5d9cbc4ff405:
FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real
0.014s user 0.001s sys 0.000s Metrics:
{"bytes_written":476,"delete_count":0,"fdatasync":3,"fdatasync_us":12119,"lbm_write_time_us":23,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":21,"threads_started":1,"update_count":3,"wal-append.queue_time_us":63}
I0523 23:30:44.493170 5907 maintenance_manager.cc:382] P
a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I0523 23:30:44.498075 5906 maintenance_manager.cc:603] P
a01491afc9f24dc3adef5d9cbc4ff405:
MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real
0.005s user 0.001s sys 0.000s Metrics:
{"cfile_cache_hit":1,"cfile_cache_hit_bytes":29,"cfile_cache_miss":3,"cfile_cache_miss_bytes":1283,"cfile_init":1,"delete_count":146,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"fdatasync":3,"fdatasync_us":3378,"lbm_read_time_us":5,"lbm_reads_lt_1ms":7,"lbm_write_time_us":11,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":16,"threads_started":1,"update_count":3}
I0524 00:00:48.336750 5917 catalog_manager.cc:2253] Servicing DeleteTable
request from {username='root'} at 192.168.2.182:42712:
table { table_name: "test2.users" } modify_external_catalogs: true
W0524 00:00:48.349134 5917 catalog_manager.cc:2296] failed to delete HMS
catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]: Not
found: failed to drop Hive Metastore table: TException - service has thrown:
NoSuchObjectException(message=hive.test2.users table not found)
{code}
!image-2022-05-24-00-11-43-309.png!
kudu cluster ksck gpmaster,gpnode1,gpnode2
{code:java}
kudu cluster ksck gpmaster,gpnode1,gpnode2
W0524 14:07:28.912752 13740 ksck_remote.cc:301] Couldn't fetch quiescing info
from tablet server 7f33079cf18941eda06a0fcc06c3afd3 (192.168.2.183:7050):
Remote error: Not authorized: unauthorized access to method: Quiesce
W0524 14:07:28.913740 13739 ksck_remote.cc:301] Couldn't fetch quiescing info
from tablet server 524c9034d0b540cab9161d44a5dab7ef (192.168.2.184:7050):
Remote error: Not authorized: unauthorized access to method: Quiesce
W0524 14:07:28.913827 13741 ksck_remote.cc:301] Couldn't fetch quiescing info
from tablet server 9ad2b92eba814e78895e645f74618bd2 (192.168.2.182:7050):
Remote error: Not authorized: unauthorized access to method: Quiesce
W0524 14:07:28.920079 13700 ksck_remote.cc:301] Couldn't fetch quiescing info
from tablet server 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211:7050):
Remote error: Not authorized: unauthorized access to method: Quiesce
Master Summary
UUID | Address | Status
----------------------------------+----------+---------
2bc3b2318ca640a78a99fcbe4d058a9f | gpmaster | HEALTHY
a01491afc9f24dc3adef5d9cbc4ff405 | gpnode1 | HEALTHY
ccbd77b62e784359b71c07d16a909e59 | gpnode2 | HEALTHYUnusual flags for Master:
Flag | Value | Tags | Master
------------------+-------+--------+-------------------------
use_hybrid_clock | false | hidden | all 3 server(s) checkedFlags of checked
categories for Master:
Flag | Value
| Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers |
0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s)
checked
time_source | system
| all 3 server(s) checkedFlags of checked categories for Master diverging
from Tablet Server flags:
Flag | Value
| Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers |
0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s)
checked
time_source | system
| all 3 server(s) checkedTablet Server Summary
UUID | Address | Status | Location
----------------------------------+--------------------+--------------+----------
18c9a3f5aa314a2b80dfa970003e4350 | 192.168.2.211:7050 | UNAUTHORIZED | <none>
524c9034d0b540cab9161d44a5dab7ef | 192.168.2.184:7050 | UNAUTHORIZED | <none>
7f33079cf18941eda06a0fcc06c3afd3 | 192.168.2.183:7050 | UNAUTHORIZED | <none>
9ad2b92eba814e78895e645f74618bd2 | 192.168.2.182:7050 | UNAUTHORIZED |
<none>Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Error from 192.168.2.211:7050: Remote error: could not fetch all consensus
info: Not authorized: unauthorized access to method: GetConsensusState
(UNAUTHORIZED)
Error from 192.168.2.184:7050: Remote error: could not fetch all consensus
info: Not authorized: unauthorized access to method: GetConsensusState
(UNAUTHORIZED)
Error from 192.168.2.183:7050: Remote error: could not fetch all consensus
info: Not authorized: unauthorized access to method: GetConsensusState
(UNAUTHORIZED)
Error from 192.168.2.182:7050: Remote error: could not fetch all consensus
info: Not authorized: unauthorized access to method: GetConsensusState
(UNAUTHORIZED)Version Summary
Version | Servers
---------+-------------------------
1.15.0 | all 7 server(s) checkedTablet Summary
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering |
Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
$schemas | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users1 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users10 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users11 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users12 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users13 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users2 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users3 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users4 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users5 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users6 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users7 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users8 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users9 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test2.users | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test2.users1 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test2.users2 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test2.users3 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 34
Median | 35
Third Quartile | 35
Maximum | 35Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 18c9a3f5aa314a2b80dfa970003e4350 | 192.168.2.211:7050 | 10Total Count
Summary
| Total Count
----------------+-------------
Masters | 3
Tablet Servers | 4
Tables | 19
Tablets | 38
Replicas | 114==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 4 of 4 tservers were not available to
retrieve unusual flags
tserver diverged flags check error: 4 of 4 tservers were not available to
retrieve time_source category flags
Same flags have different values between masters and tablet servers for at
least one checked flag category==================
Errors:
==================
Not authorized: error fetching info from tablet servers: failed to gather info
from 4 of 4 tablet servers due to lack of admin privilegesFAILED
Not authorized: re-run ksck with administrator privileges
{code}
was:
At present, according to the patch provided by hive-16993, hive 3.1.1 is
recompiled. Now it can successfully synchronize with hive metadata and create
tables. However, when deleting tables, the log shows that the tables have been
successfully deleted, but the tables in kudu still exist and cannot be deleted。
{code:java}
USE
trino:test2> CREATE TABLE users (
-> userid int WITH (primary_key = true),
-> firstname varchar,
-> lastname varchar
-> ) WITH (
-> partition_by_hash_columns = ARRAY['userid'],
-> partition_by_hash_buckets = 2
-> );
CREATE TABLE
trino:test2> show tables;
Table
-------
users
(1 row)
trino:test2> drop table users;
DROP TABLE
trino:test2> show tables;
Table
-------
users
(1 row)Query 20220523_152932_00147_ytvpx, FINISHED, 3 nodes
Splits: 20 total, 20 done (100.00%)
0.23 [1 rows, 20B] [4 rows/s, 86B/s]
trino:test2> drop table users;
Query 20220523_152946_00148_ytvpx failed: failed to drop Hive Metastore table:
TException - service has thrown: NoSuchObjectException(message=hive.test2.users
table not found) {code}
The metadata of hive has been successfully deleted when I delete it for the
first time. When I delete it for the second time, it is reported that the
metadata does not exist because it has been deleted, but the table still exists
in kudu. Use the command:
{code:java}
kudu table delete gpnode1 test2.users{code}
error message：
{code:java}
# kudu table delete gpnode1 test2.users
# Not found: failed to drop Hive Metastore table: TException - service has
thrown: NoSuchObjectException(message=hive.test2.users table not found) {code}
log：
{code:java}
W0523 23:21:52.950664 11595 master_path_handlers.cc:837] Unable to fetch master
addresses: OK
W0523 23:22:31.867633 11595 master_path_handlers.cc:837] Unable to fetch master
addresses: OK
I0523 23:28:41.818248 5917 catalog_manager.cc:1681] Servicing CreateTable
request from {username='presto'} at 192.168.2.182:36402:
name: "test2.users"
schema {
columns {
name: "userid"
type: INT32
is_key: true
is_nullable: false
cfile_block_size: 0
}
columns {
name: "firstname"
type: STRING
is_key: false
is_nullable: false
cfile_block_size: 0
}
columns {
name: "lastname"
type: STRING
is_key: false
is_nullable: false
cfile_block_size: 0
}
}
partition_schema {
hash_bucket_schemas {
columns {
name: "userid"
}
num_buckets: 2
seed: 0
}
}
I0523 23:28:41.877279 5917 catalog_manager.cc:1963] created HMS catalog entry
for table [id=3a50aa7d5aa6416f8ab5587ed16bc85a]
I0523 23:28:42.091735 5907 maintenance_manager.cc:382] P
a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
FlushMRSOp(00000000000000000000000000000000): perf score=0.302975
I0523 23:28:42.109527 5906 maintenance_manager.cc:603] P
a01491afc9f24dc3adef5d9cbc4ff405: FlushMRSOp(00000000000000000000000000000000)
complete. Timing: real 0.018s user 0.003s sys 0.003s Metrics:
{"bytes_written":5660,"cfile_init":1,"dirs.queue_time_us":90,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":667,"drs_written":1,"fdatasync":13,"fdatasync_us":11633,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4,"lbm_write_time_us":133,"lbm_writes_lt_1ms":27,"rows_written":3,"thread_start_us":59,"threads_started":1}
I0523 23:28:42.109880 5907 maintenance_manager.cc:382] P
a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
CompactRowSetsOp(00000000000000000000000000000000): perf score=0.212344
I0523 23:28:42.133885 5906 maintenance_manager.cc:603] P
a01491afc9f24dc3adef5d9cbc4ff405:
CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real
0.024s user 0.008s sys 0.004s Metrics:
{"bytes_written":213198,"cfile_cache_hit":13,"cfile_cache_hit_bytes":206138,"cfile_cache_miss":35,"cfile_cache_miss_bytes":225520,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":502,"dirs.run_cpu_time_us":810,"dirs.run_wall_time_us":950,"drs_written":1,"fdatasync":15,"fdatasync_us":12534,"lbm_read_time_us":662,"lbm_reads_lt_1ms":75,"lbm_write_time_us":530,"lbm_writes_lt_1ms":32,"num_input_rowsets":2,"rows_written":209,"spinlock_wait_cycles":2944,"thread_start_us":397,"threads_started":4}
I0523 23:28:44.408016 5917 catalog_manager.cc:4733] T
273e6e3223f14ea690621e40c53c8c0e P 18c9a3f5aa314a2b80dfa970003e4350 reported
cstate change: term changed from 0 to 1, leader changed from <none> to
18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211). New cstate: current_term: 1
leader_uuid: "18c9a3f5aa314a2b80dfa970003e4350" committed_config { opid_index:
-1 OBSOLETE_local: false peers { permanent_uuid:
"524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host:
"192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } peers
{ permanent_uuid: "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER
last_known_addr { host: "192.168.2.211" port: 7050 } health_report {
overall_health: HEALTHY } } peers { permanent_uuid:
"9ad2b92eba814e78895e645f74618bd2" member_type: VOTER last_known_addr { host:
"192.168.2.182" port: 7050 } health_report { overall_health: UNKNOWN } } }
I0523 23:28:59.145552 5917 catalog_manager.cc:4733] T
e0d2911469d147609abcdcebffacc7f1 P 18c9a3f5aa314a2b80dfa970003e4350 reported
cstate change: term changed from 0 to 1, leader changed from <none> to
18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211). New cstate: current_term: 1
leader_uuid: "18c9a3f5aa314a2b80dfa970003e4350" committed_config { opid_index:
-1 OBSOLETE_local: false peers { permanent_uuid:
"18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host:
"192.168.2.211" port: 7050 } health_report { overall_health: HEALTHY } } peers
{ permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER
last_known_addr { host: "192.168.2.182" port: 7050 } health_report {
overall_health: UNKNOWN } } peers { permanent_uuid:
"524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host:
"192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } }
I0523 23:29:28.640434 5917 catalog_manager.cc:2253] Servicing DeleteTable
request from {username='presto'} at 192.168.2.182:36402:
table { table_name: "test2.users" }
I0523 23:29:28.706779 5917 catalog_manager.cc:2293] deleted HMS catalog entry
for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]
I0523 23:29:33.955879 5917 catalog_manager.cc:4733] T
e0d2911469d147609abcdcebffacc7f1 P 9ad2b92eba814e78895e645f74618bd2 reported
cstate change: term changed from 1 to 2, leader changed from
18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211) to
9ad2b92eba814e78895e645f74618bd2 (192.168.2.182). New cstate: current_term: 2
leader_uuid: "9ad2b92eba814e78895e645f74618bd2" committed_config { opid_index:
-1 OBSOLETE_local: false peers { permanent_uuid:
"18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host:
"192.168.2.211" port: 7050 } health_report { overall_health: UNKNOWN } } peers
{ permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER
last_known_addr { host: "192.168.2.182" port: 7050 } health_report {
overall_health: HEALTHY } } peers { permanent_uuid:
"524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host:
"192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } }
I0523 23:29:46.440908 5917 catalog_manager.cc:2253] Servicing DeleteTable
request from {username='presto'} at 192.168.2.182:36402:
table { table_name: "test2.users" }
W0523 23:29:46.446835 5917 catalog_manager.cc:2296] failed to delete HMS
catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]: Not
found: failed to drop Hive Metastore table: TException - service has thrown:
NoSuchObjectException(message=hive.test2.users table not found)
I0523 23:30:44.479030 5907 maintenance_manager.cc:382] P
a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033352
I0523 23:30:44.493083 5906 maintenance_manager.cc:603] P
a01491afc9f24dc3adef5d9cbc4ff405:
FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real
0.014s user 0.001s sys 0.000s Metrics:
{"bytes_written":476,"delete_count":0,"fdatasync":3,"fdatasync_us":12119,"lbm_write_time_us":23,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":21,"threads_started":1,"update_count":3,"wal-append.queue_time_us":63}
I0523 23:30:44.493170 5907 maintenance_manager.cc:382] P
a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I0523 23:30:44.498075 5906 maintenance_manager.cc:603] P
a01491afc9f24dc3adef5d9cbc4ff405:
MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real
0.005s user 0.001s sys 0.000s Metrics:
{"cfile_cache_hit":1,"cfile_cache_hit_bytes":29,"cfile_cache_miss":3,"cfile_cache_miss_bytes":1283,"cfile_init":1,"delete_count":146,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"fdatasync":3,"fdatasync_us":3378,"lbm_read_time_us":5,"lbm_reads_lt_1ms":7,"lbm_write_time_us":11,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":16,"threads_started":1,"update_count":3}
I0524 00:00:48.336750 5917 catalog_manager.cc:2253] Servicing DeleteTable
request from {username='root'} at 192.168.2.182:42712:
table { table_name: "test2.users" } modify_external_catalogs: true
W0524 00:00:48.349134 5917 catalog_manager.cc:2296] failed to delete HMS
catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]: Not
found: failed to drop Hive Metastore table: TException - service has thrown:
NoSuchObjectException(message=hive.test2.users table not found)
{code}
!image-2022-05-24-00-11-43-309.png!
kudu cluster ksck gpnode1
{code:java}
W0524 11:24:37.228371 29382 ksck_remote.cc:301] Couldn't fetch quiescing info
from tablet server 7f33079cf18941eda06a0fcc06c3afd3 (192.168.2.183:7050):
Remote error: Not authorized: unauthorized access to method: Quiesce
W0524 11:24:37.231554 29381 ksck_remote.cc:301] Couldn't fetch quiescing info
from tablet server 524c9034d0b540cab9161d44a5dab7ef (192.168.2.184:7050):
Remote error: Not authorized: unauthorized access to method: Quiesce
W0524 11:24:37.231554 29383 ksck_remote.cc:301] Couldn't fetch quiescing info
from tablet server 9ad2b92eba814e78895e645f74618bd2 (192.168.2.182:7050):
Remote error: Not authorized: unauthorized access to method: Quiesce
W0524 11:24:37.236119 29349 ksck_remote.cc:301] Couldn't fetch quiescing info
from tablet server 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211:7050):
Remote error: Not authorized: unauthorized access to method: Quiesce
Master Summary
UUID | Address | Status
----------------------------------+---------+---------
a01491afc9f24dc3adef5d9cbc4ff405 | gpnode1 | HEALTHYUnusual flags for Master:
Flag | Value | Tags | Master
------------------+-------+--------+-------------------------
use_hybrid_clock | false | hidden | all 1 server(s) checkedFlags of checked
categories for Master:
Flag | Value
| Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers |
0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s)
checked
time_source | system
| all 1 server(s) checkedFlags of checked categories for Master diverging
from Tablet Server flags:
Flag | Value
| Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers |
0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s)
checked
time_source | system
| all 1 server(s) checkedTablet Server Summary
UUID | Address | Status | Location
----------------------------------+--------------------+--------------+----------
18c9a3f5aa314a2b80dfa970003e4350 | 192.168.2.211:7050 | UNAUTHORIZED | <none>
524c9034d0b540cab9161d44a5dab7ef | 192.168.2.184:7050 | UNAUTHORIZED | <none>
7f33079cf18941eda06a0fcc06c3afd3 | 192.168.2.183:7050 | UNAUTHORIZED | <none>
9ad2b92eba814e78895e645f74618bd2 | 192.168.2.182:7050 | UNAUTHORIZED |
<none>Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Error from 192.168.2.211:7050: Remote error: could not fetch all consensus
info: Not authorized: unauthorized access to method: GetConsensusState
(UNAUTHORIZED)
Error from 192.168.2.184:7050: Remote error: could not fetch all consensus
info: Not authorized: unauthorized access to method: GetConsensusState
(UNAUTHORIZED)
Error from 192.168.2.183:7050: Remote error: could not fetch all consensus
info: Not authorized: unauthorized access to method: GetConsensusState
(UNAUTHORIZED)
Error from 192.168.2.182:7050: Remote error: could not fetch all consensus
info: Not authorized: unauthorized access to method: GetConsensusState
(UNAUTHORIZED)Version Summary
Version | Servers
---------+-------------------------
1.15.0 | all 5 server(s) checkedTablet Summary
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering |
Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
$schemas | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users1 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users10 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users11 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users12 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users13 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users2 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users3 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users4 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users5 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users6 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users7 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users8 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test1.users9 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test2.users | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test2.users1 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test2.users2 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0
test2.users3 | 3 | HEALTHY | 2 | 2 | 0 | 0
| 0Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 34
Median | 35
Third Quartile | 35
Maximum | 35Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | 18c9a3f5aa314a2b80dfa970003e4350 | 192.168.2.211:7050 | 10Total Count
Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 19
Tablets | 38
Replicas | 114==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 4 of 4 tservers were not available to
retrieve unusual flags
tserver diverged flags check error: 4 of 4 tservers were not available to
retrieve time_source category flags
Same flags have different values between masters and tablet servers for at
least one checked flag category==================
Errors:
==================
Not authorized: error fetching info from tablet servers: failed to gather info
from 4 of 4 tablet servers due to lack of admin privilegesFAILED
Not authorized: re-run ksck with administrator privileges
{code}
> drop table failed
> -----------------
>
> Key: KUDU-3370
> URL: https://issues.apache.org/jira/browse/KUDU-3370
> Project: Kudu
> Issue Type: Bug
> Components: master, tserver
> Affects Versions: 1.15.0
> Environment: hdp :3.2.3
> hive :3.1.1
> kudu :1.15.0
> trino:369-SNAPSHOT
> Reporter: liu
> Priority: Critical
> Fix For: NA
>
> Attachments: image-2022-05-24-00-11-43-309.png
>
>
> At present, according to the patch provided by hive-16993, hive 3.1.1 is
> recompiled. Now it can successfully synchronize with hive metadata and create
> tables. However, when deleting tables, the log shows that the tables have
> been successfully deleted, but the tables in kudu still exist and cannot be
> deleted。
> {code:java}
> USE
> trino:test2> CREATE TABLE users (
> -> userid int WITH (primary_key = true),
> -> firstname varchar,
> -> lastname varchar
> -> ) WITH (
> -> partition_by_hash_columns = ARRAY['userid'],
> -> partition_by_hash_buckets = 2
> -> );
> CREATE TABLE
> trino:test2> show tables;
> Table
> -------
> users
> (1 row)
> trino:test2> drop table users;
> DROP TABLE
> trino:test2> show tables;
> Table
> -------
> users
> (1 row)Query 20220523_152932_00147_ytvpx, FINISHED, 3 nodes
> Splits: 20 total, 20 done (100.00%)
> 0.23 [1 rows, 20B] [4 rows/s, 86B/s]
> trino:test2> drop table users;
> Query 20220523_152946_00148_ytvpx failed: failed to drop Hive Metastore
> table: TException - service has thrown:
> NoSuchObjectException(message=hive.test2.users table not found) {code}
> The metadata of hive has been successfully deleted when I delete it for the
> first time. When I delete it for the second time, it is reported that the
> metadata does not exist because it has been deleted, but the table still
> exists in kudu. Use the command:
> {code:java}
> kudu table delete gpnode1 test2.users{code}
> error message：
> {code:java}
> # kudu table delete gpnode1 test2.users
> # Not found: failed to drop Hive Metastore table: TException - service has
> thrown: NoSuchObjectException(message=hive.test2.users table not found) {code}
> log：
> {code:java}
> W0523 23:21:52.950664 11595 master_path_handlers.cc:837] Unable to fetch
> master addresses: OK
> W0523 23:22:31.867633 11595 master_path_handlers.cc:837] Unable to fetch
> master addresses: OK
> I0523 23:28:41.818248 5917 catalog_manager.cc:1681] Servicing CreateTable
> request from {username='presto'} at 192.168.2.182:36402:
> name: "test2.users"
> schema {
> columns {
> name: "userid"
> type: INT32
> is_key: true
> is_nullable: false
> cfile_block_size: 0
> }
> columns {
> name: "firstname"
> type: STRING
> is_key: false
> is_nullable: false
> cfile_block_size: 0
> }
> columns {
> name: "lastname"
> type: STRING
> is_key: false
> is_nullable: false
> cfile_block_size: 0
> }
> }
> partition_schema {
> hash_bucket_schemas {
> columns {
> name: "userid"
> }
> num_buckets: 2
> seed: 0
> }
> }
> I0523 23:28:41.877279 5917 catalog_manager.cc:1963] created HMS catalog
> entry for table [id=3a50aa7d5aa6416f8ab5587ed16bc85a]
> I0523 23:28:42.091735 5907 maintenance_manager.cc:382] P
> a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
> FlushMRSOp(00000000000000000000000000000000): perf score=0.302975
> I0523 23:28:42.109527 5906 maintenance_manager.cc:603] P
> a01491afc9f24dc3adef5d9cbc4ff405:
> FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.018s
> user 0.003s sys 0.003s Metrics:
> {"bytes_written":5660,"cfile_init":1,"dirs.queue_time_us":90,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":667,"drs_written":1,"fdatasync":13,"fdatasync_us":11633,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4,"lbm_write_time_us":133,"lbm_writes_lt_1ms":27,"rows_written":3,"thread_start_us":59,"threads_started":1}
> I0523 23:28:42.109880 5907 maintenance_manager.cc:382] P
> a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
> CompactRowSetsOp(00000000000000000000000000000000): perf score=0.212344
> I0523 23:28:42.133885 5906 maintenance_manager.cc:603] P
> a01491afc9f24dc3adef5d9cbc4ff405:
> CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real
> 0.024s user 0.008s sys 0.004s Metrics:
> {"bytes_written":213198,"cfile_cache_hit":13,"cfile_cache_hit_bytes":206138,"cfile_cache_miss":35,"cfile_cache_miss_bytes":225520,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":502,"dirs.run_cpu_time_us":810,"dirs.run_wall_time_us":950,"drs_written":1,"fdatasync":15,"fdatasync_us":12534,"lbm_read_time_us":662,"lbm_reads_lt_1ms":75,"lbm_write_time_us":530,"lbm_writes_lt_1ms":32,"num_input_rowsets":2,"rows_written":209,"spinlock_wait_cycles":2944,"thread_start_us":397,"threads_started":4}
> I0523 23:28:44.408016 5917 catalog_manager.cc:4733] T
> 273e6e3223f14ea690621e40c53c8c0e P 18c9a3f5aa314a2b80dfa970003e4350 reported
> cstate change: term changed from 0 to 1, leader changed from <none> to
> 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211). New cstate: current_term: 1
> leader_uuid: "18c9a3f5aa314a2b80dfa970003e4350" committed_config {
> opid_index: -1 OBSOLETE_local: false peers { permanent_uuid:
> "524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host:
> "192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } }
> peers { permanent_uuid: "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER
> last_known_addr { host: "192.168.2.211" port: 7050 } health_report {
> overall_health: HEALTHY } } peers { permanent_uuid:
> "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER last_known_addr { host:
> "192.168.2.182" port: 7050 } health_report { overall_health: UNKNOWN } } }
> I0523 23:28:59.145552 5917 catalog_manager.cc:4733] T
> e0d2911469d147609abcdcebffacc7f1 P 18c9a3f5aa314a2b80dfa970003e4350 reported
> cstate change: term changed from 0 to 1, leader changed from <none> to
> 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211). New cstate: current_term: 1
> leader_uuid: "18c9a3f5aa314a2b80dfa970003e4350" committed_config {
> opid_index: -1 OBSOLETE_local: false peers { permanent_uuid:
> "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host:
> "192.168.2.211" port: 7050 } health_report { overall_health: HEALTHY } }
> peers { permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER
> last_known_addr { host: "192.168.2.182" port: 7050 } health_report {
> overall_health: UNKNOWN } } peers { permanent_uuid:
> "524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host:
> "192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } }
> I0523 23:29:28.640434 5917 catalog_manager.cc:2253] Servicing DeleteTable
> request from {username='presto'} at 192.168.2.182:36402:
> table { table_name: "test2.users" }
> I0523 23:29:28.706779 5917 catalog_manager.cc:2293] deleted HMS catalog
> entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]
> I0523 23:29:33.955879 5917 catalog_manager.cc:4733] T
> e0d2911469d147609abcdcebffacc7f1 P 9ad2b92eba814e78895e645f74618bd2 reported
> cstate change: term changed from 1 to 2, leader changed from
> 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211) to
> 9ad2b92eba814e78895e645f74618bd2 (192.168.2.182). New cstate: current_term: 2
> leader_uuid: "9ad2b92eba814e78895e645f74618bd2" committed_config {
> opid_index: -1 OBSOLETE_local: false peers { permanent_uuid:
> "18c9a3f5aa314a2b80dfa970003e4350" member_type: VOTER last_known_addr { host:
> "192.168.2.211" port: 7050 } health_report { overall_health: UNKNOWN } }
> peers { permanent_uuid: "9ad2b92eba814e78895e645f74618bd2" member_type: VOTER
> last_known_addr { host: "192.168.2.182" port: 7050 } health_report {
> overall_health: HEALTHY } } peers { permanent_uuid:
> "524c9034d0b540cab9161d44a5dab7ef" member_type: VOTER last_known_addr { host:
> "192.168.2.184" port: 7050 } health_report { overall_health: UNKNOWN } } }
> I0523 23:29:46.440908 5917 catalog_manager.cc:2253] Servicing DeleteTable
> request from {username='presto'} at 192.168.2.182:36402:
> table { table_name: "test2.users" }
> W0523 23:29:46.446835 5917 catalog_manager.cc:2296] failed to delete HMS
> catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]:
> Not found: failed to drop Hive Metastore table: TException - service has
> thrown: NoSuchObjectException(message=hive.test2.users table not found)
> I0523 23:30:44.479030 5907 maintenance_manager.cc:382] P
> a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
> FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033352
> I0523 23:30:44.493083 5906 maintenance_manager.cc:603] P
> a01491afc9f24dc3adef5d9cbc4ff405:
> FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing:
> real 0.014s user 0.001s sys 0.000s Metrics:
> {"bytes_written":476,"delete_count":0,"fdatasync":3,"fdatasync_us":12119,"lbm_write_time_us":23,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":21,"threads_started":1,"update_count":3,"wal-append.queue_time_us":63}
> I0523 23:30:44.493170 5907 maintenance_manager.cc:382] P
> a01491afc9f24dc3adef5d9cbc4ff405: Scheduling
> MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
> I0523 23:30:44.498075 5906 maintenance_manager.cc:603] P
> a01491afc9f24dc3adef5d9cbc4ff405:
> MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing:
> real 0.005s user 0.001s sys 0.000s Metrics:
> {"cfile_cache_hit":1,"cfile_cache_hit_bytes":29,"cfile_cache_miss":3,"cfile_cache_miss_bytes":1283,"cfile_init":1,"delete_count":146,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"fdatasync":3,"fdatasync_us":3378,"lbm_read_time_us":5,"lbm_reads_lt_1ms":7,"lbm_write_time_us":11,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":16,"threads_started":1,"update_count":3}
> I0524 00:00:48.336750 5917 catalog_manager.cc:2253] Servicing DeleteTable
> request from {username='root'} at 192.168.2.182:42712:
> table { table_name: "test2.users" } modify_external_catalogs: true
> W0524 00:00:48.349134 5917 catalog_manager.cc:2296] failed to delete HMS
> catalog entry for table test2.users [id=3a50aa7d5aa6416f8ab5587ed16bc85a]:
> Not found: failed to drop Hive Metastore table: TException - service has
> thrown: NoSuchObjectException(message=hive.test2.users table not found)
> {code}
>
>
> !image-2022-05-24-00-11-43-309.png!
>
> kudu cluster ksck gpmaster,gpnode1,gpnode2
> {code:java}
> kudu cluster ksck gpmaster,gpnode1,gpnode2
> W0524 14:07:28.912752 13740 ksck_remote.cc:301] Couldn't fetch quiescing info
> from tablet server 7f33079cf18941eda06a0fcc06c3afd3 (192.168.2.183:7050):
> Remote error: Not authorized: unauthorized access to method: Quiesce
> W0524 14:07:28.913740 13739 ksck_remote.cc:301] Couldn't fetch quiescing info
> from tablet server 524c9034d0b540cab9161d44a5dab7ef (192.168.2.184:7050):
> Remote error: Not authorized: unauthorized access to method: Quiesce
> W0524 14:07:28.913827 13741 ksck_remote.cc:301] Couldn't fetch quiescing info
> from tablet server 9ad2b92eba814e78895e645f74618bd2 (192.168.2.182:7050):
> Remote error: Not authorized: unauthorized access to method: Quiesce
> W0524 14:07:28.920079 13700 ksck_remote.cc:301] Couldn't fetch quiescing info
> from tablet server 18c9a3f5aa314a2b80dfa970003e4350 (192.168.2.211:7050):
> Remote error: Not authorized: unauthorized access to method: Quiesce
> Master Summary
> UUID | Address | Status
> ----------------------------------+----------+---------
> 2bc3b2318ca640a78a99fcbe4d058a9f | gpmaster | HEALTHY
> a01491afc9f24dc3adef5d9cbc4ff405 | gpnode1 | HEALTHY
> ccbd77b62e784359b71c07d16a909e59 | gpnode2 | HEALTHYUnusual flags for
> Master:
> Flag | Value | Tags | Master
> ------------------+-------+--------+-------------------------
> use_hybrid_clock | false | hidden | all 3 server(s) checkedFlags of checked
> categories for Master:
> Flag | Value
> | Master
> ---------------------+-------------------------------------------------------------+-------------------------
> builtin_ntp_servers |
> 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s)
> checked
> time_source | system
> | all 3 server(s) checkedFlags of checked categories for Master
> diverging from Tablet Server flags:
> Flag | Value
> | Master
> ---------------------+-------------------------------------------------------------+-------------------------
> builtin_ntp_servers |
> 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s)
> checked
> time_source | system
> | all 3 server(s) checkedTablet Server Summary
> UUID | Address | Status |
> Location
> ----------------------------------+--------------------+--------------+----------
> 18c9a3f5aa314a2b80dfa970003e4350 | 192.168.2.211:7050 | UNAUTHORIZED | <none>
> 524c9034d0b540cab9161d44a5dab7ef | 192.168.2.184:7050 | UNAUTHORIZED | <none>
> 7f33079cf18941eda06a0fcc06c3afd3 | 192.168.2.183:7050 | UNAUTHORIZED | <none>
> 9ad2b92eba814e78895e645f74618bd2 | 192.168.2.182:7050 | UNAUTHORIZED |
> <none>Tablet Server Location Summary
> Location | Count
> ----------+---------
> <none> | 4
> Error from 192.168.2.211:7050: Remote error: could not fetch all consensus
> info: Not authorized: unauthorized access to method: GetConsensusState
> (UNAUTHORIZED)
> Error from 192.168.2.184:7050: Remote error: could not fetch all consensus
> info: Not authorized: unauthorized access to method: GetConsensusState
> (UNAUTHORIZED)
> Error from 192.168.2.183:7050: Remote error: could not fetch all consensus
> info: Not authorized: unauthorized access to method: GetConsensusState
> (UNAUTHORIZED)
> Error from 192.168.2.182:7050: Remote error: could not fetch all consensus
> info: Not authorized: unauthorized access to method: GetConsensusState
> (UNAUTHORIZED)Version Summary
> Version | Servers
> ---------+-------------------------
> 1.15.0 | all 7 server(s) checkedTablet Summary
> Summary by table
> Name | RF | Status | Total Tablets | Healthy | Recovering |
> Under-replicated | Unavailable
> ---------------+----+---------+---------------+---------+------------+------------------+-------------
> $schemas | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users1 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users10 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users11 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users12 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users13 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users2 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users3 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users4 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users5 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users6 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users7 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users8 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test1.users9 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test2.users | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test2.users1 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test2.users2 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0
> test2.users3 | 3 | HEALTHY | 2 | 2 | 0 | 0
> | 0Tablet Replica Count Summary
> Statistic | Replica Count
> ----------------+---------------
> Minimum | 10
> First Quartile | 34
> Median | 35
> Third Quartile | 35
> Maximum | 35Tablet Replica Count Outliers
> Type | UUID | Host | Replica Count
> -------+----------------------------------+--------------------+---------------
> Small | 18c9a3f5aa314a2b80dfa970003e4350 | 192.168.2.211:7050 | 10Total
> Count Summary
> | Total Count
> ----------------+-------------
> Masters | 3
> Tablet Servers | 4
> Tables | 19
> Tablets | 38
> Replicas | 114==================
> Warnings:
> ==================
> Some masters have unsafe, experimental, or hidden flags set
> tserver unusual flags check error: 4 of 4 tservers were not available to
> retrieve unusual flags
> tserver diverged flags check error: 4 of 4 tservers were not available to
> retrieve time_source category flags
> Same flags have different values between masters and tablet servers for at
> least one checked flag category==================
> Errors:
> ==================
> Not authorized: error fetching info from tablet servers: failed to gather
> info from 4 of 4 tablet servers due to lack of admin privilegesFAILED
> Not authorized: re-run ksck with administrator privileges
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
