[
https://issues.apache.org/jira/browse/IMPALA-11187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644441#comment-17644441
]
Michael Smith commented on IMPALA-11187:
----------------------------------------
I looked through Kudu and Impala logs. I don't see anything obvious, although
in the middle of this query Kudu tablet servers are doing
{code}
I1207 02:40:47.913925 17518 tablet_service.cc:1512] Processing DeleteTablet for
tablet a010773aad414657975ec8fcd20b84fc with delete_type TABLET_DATA_DELETED
(Table deleted at 2022-12-07 02:40:47 UTC) from {username='ubuntu'} at
127.0.0.1:48856
I1207 02:40:47.913942 17502 tablet_service.cc:1512] Processing DeleteTablet for
tablet 63d011a0c5a340f2909efd885c06b0ed with delete_type TABLET_DATA_DELETED
(Table deleted at 2022-12-07 02:40:47 UTC) from {username='ubuntu'} at
127.0.0.1:48856
I1207 02:40:47.913933 17515 tablet_service.cc:1512] Processing DeleteTablet for
tablet 387108e0cf094873b8d61319178fcfe1 with delete_type TABLET_DATA_DELETED
(Table deleted at 2022-12-07 02:40:47 UTC) from {username='ubuntu'} at
127.0.0.1:48856
I1207 02:40:47.914009 17502 tablet_service.cc:1512] Processing DeleteTablet for
tablet 377b32f8e9c542338249be052aae9d20 with delete_type TABLET_DATA_DELETED
(Table deleted at 2022-12-07 02:40:47 UTC) from {username='ubuntu'} at
127.0.0.1:48856
I1207 02:40:47.914029 17515 tablet_service.cc:1512] Processing DeleteTablet for
tablet 3dac2fd02f6045b3aa01d06e00d7fdf2 with delete_type TABLET_DATA_DELETED
(Table deleted at 2022-12-07 02:40:47 UTC) from {username='ubuntu'} at
127.0.0.1:48856
I1207 02:40:47.914047 17502 tablet_service.cc:1512] Processing DeleteTablet for
tablet fa987644f7554383a1a9b6984e54cfa2 with delete_type TABLET_DATA_DELETED
(Table deleted at 2022-12-07 02:40:47 UTC) from {username='ubuntu'} at
127.0.0.1:48856
I1207 02:40:47.914064 17518 tablet_service.cc:1512] Processing DeleteTablet for
tablet b4318238b0e94b718881e5c6d0afd0e5 with delete_type TABLET_DATA_DELETED
(Table deleted at 2022-12-07 02:40:47 UTC) from {username='ubuntu'} at
127.0.0.1:48856
I1207 02:40:47.914077 17515 tablet_service.cc:1512] Processing DeleteTablet for
tablet daf8246510f14af1bab937551059d615 with delete_type TABLET_DATA_DELETED
(Table deleted at 2022-12-07 02:40:47 UTC) from {username='ubuntu'} at
127.0.0.1:48856
I1207 02:40:47.915189 8717 tablet_replica.cc:330] T
63d011a0c5a340f2909efd885c06b0ed P 8292ec3a6791482b847b5728af5c3012: stopping
tablet replica
I1207 02:40:47.915246 8717 raft_consensus.cc:2229] T
63d011a0c5a340f2909efd885c06b0ed P 8292ec3a6791482b847b5728af5c3012 [term 1
LEADER]: Raft consensus shutting down.
I1207 02:40:47.915292 8717 raft_consensus.cc:2258] T
63d011a0c5a340f2909efd885c06b0ed P 8292ec3a6791482b847b5728af5c3012 [term 1
FOLLOWER]: Raft consensus is shut down!
I1207 02:40:47.915372 8717 ts_tablet_manager.cc:1824] T
63d011a0c5a340f2909efd885c06b0ed P 8292ec3a6791482b847b5728af5c3012: Deleting
tablet data with delete state TABLET_DATA_DELETED
I1207 02:40:47.916625 8717 ts_tablet_manager.cc:1837] T
63d011a0c5a340f2909efd885c06b0ed P 8292ec3a6791482b847b5728af5c3012: tablet
deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I1207 02:40:47.916648 8717 log.cc:1192] T 63d011a0c5a340f2909efd885c06b0ed P
8292ec3a6791482b847b5728af5c3012: Deleting WAL directory at
/home/ubuntu/Impala/testdata/cluster/cdh7/node-3/var/lib/kudu/ts/wal/wals/63d011a0c5a340f2909efd885c06b0ed
I1207 02:40:47.916810 8717 ts_tablet_manager.cc:1858] T
63d011a0c5a340f2909efd885c06b0ed P 8292ec3a6791482b847b5728af5c3012: Deleting
consensus metadata
I1207 02:40:47.916927 8717 tablet_replica.cc:330] T
a010773aad414657975ec8fcd20b84fc P 8292ec3a6791482b847b5728af5c3012: stopping
tablet replica
I1207 02:40:47.916961 8717 raft_consensus.cc:2229] T
a010773aad414657975ec8fcd20b84fc P 8292ec3a6791482b847b5728af5c3012 [term 1
FOLLOWER]: Raft consensus shutting down.
I1207 02:40:47.916975 8717 raft_consensus.cc:2258] T
a010773aad414657975ec8fcd20b84fc P 8292ec3a6791482b847b5728af5c3012 [term 1
FOLLOWER]: Raft consensus is shut down!
I1207 02:40:47.917044 8717 ts_tablet_manager.cc:1824] T
a010773aad414657975ec8fcd20b84fc P 8292ec3a6791482b847b5728af5c3012: Deleting
tablet data with delete state TABLET_DATA_DELETED
I1207 02:40:47.917963 8717 ts_tablet_manager.cc:1837] T
a010773aad414657975ec8fcd20b84fc P 8292ec3a6791482b847b5728af5c3012: tablet
deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I1207 02:40:47.917980 8717 log.cc:1192] T a010773aad414657975ec8fcd20b84fc P
8292ec3a6791482b847b5728af5c3012: Deleting WAL directory at
/home/ubuntu/Impala/testdata/cluster/cdh7/node-3/var/lib/kudu/ts/wal/wals/a010773aad414657975ec8fcd20b84fc
I1207 02:40:47.918067 8717 ts_tablet_manager.cc:1858] T
a010773aad414657975ec8fcd20b84fc P 8292ec3a6791482b847b5728af5c3012: Deleting
consensus metadata
I1207 02:40:47.918195 8717 tablet_replica.cc:330] T
387108e0cf094873b8d61319178fcfe1 P 8292ec3a6791482b847b5728af5c3012: stopping
tablet replica
I1207 02:40:47.918227 8717 raft_consensus.cc:2229] T
387108e0cf094873b8d61319178fcfe1 P 8292ec3a6791482b847b5728af5c3012 [term 1
FOLLOWER]: Raft consensus shutting down.
I1207 02:40:47.918239 8717 raft_consensus.cc:2258] T
387108e0cf094873b8d61319178fcfe1 P 8292ec3a6791482b847b5728af5c3012 [term 1
FOLLOWER]: Raft consensus is shut down!
I1207 02:40:47.918305 8717 ts_tablet_manager.cc:1824] T
387108e0cf094873b8d61319178fcfe1 P 8292ec3a6791482b847b5728af5c3012: Deleting
tablet data with delete state TABLET_DATA_DELETED
I1207 02:40:47.919057 8717 ts_tablet_manager.cc:1837] T
387108e0cf094873b8d61319178fcfe1 P 8292ec3a6791482b847b5728af5c3012: tablet
deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I1207 02:40:47.919075 8717 log.cc:1192] T 387108e0cf094873b8d61319178fcfe1 P
8292ec3a6791482b847b5728af5c3012: Deleting WAL directory at
/home/ubuntu/Impala/testdata/cluster/cdh7/node-3/var/lib/kudu/ts/wal/wals/387108e0cf094873b8d61319178fcfe1
I1207 02:40:47.919144 8717 ts_tablet_manager.cc:1858] T
387108e0cf094873b8d61319178fcfe1 P 8292ec3a6791482b847b5728af5c3012: Deleting
consensus metadata
I1207 02:40:47.919350 8717 tablet_replica.cc:330] T
377b32f8e9c542338249be052aae9d20 P 8292ec3a6791482b847b5728af5c3012: stopping
tablet replica
I1207 02:40:47.919378 8717 raft_consensus.cc:2229] T
377b32f8e9c542338249be052aae9d20 P 8292ec3a6791482b847b5728af5c3012 [term 1
FOLLOWER]: Raft consensus shutting down.
I1207 02:40:47.919389 8717 raft_consensus.cc:2258] T
377b32f8e9c542338249be052aae9d20 P 8292ec3a6791482b847b5728af5c3012 [term 1
FOLLOWER]: Raft consensus is shut down!
I1207 02:40:47.919442 8717 ts_tablet_manager.cc:1824] T
377b32f8e9c542338249be052aae9d20 P 8292ec3a6791482b847b5728af5c3012: Deleting
tablet data with delete state TABLET_DATA_DELETED
I1207 02:40:47.920148 8717 ts_tablet_manager.cc:1837] T
377b32f8e9c542338249be052aae9d20 P 8292ec3a6791482b847b5728af5c3012: tablet
deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I1207 02:40:47.920166 8717 log.cc:1192] T 377b32f8e9c542338249be052aae9d20 P
8292ec3a6791482b847b5728af5c3012: Deleting WAL directory at
/home/ubuntu/Impala/testdata/cluster/cdh7/node-3/var/lib/kudu/ts/wal/wals/377b32f8e9c542338249be052aae9d20
I1207 02:40:47.920233 8717 ts_tablet_manager.cc:1858] T
377b32f8e9c542338249be052aae9d20 P 8292ec3a6791482b847b5728af5c3012: Deleting
consensus metadata
I1207 02:40:47.920531 8717 tablet_replica.cc:330] T
3dac2fd02f6045b3aa01d06e00d7fdf2 P 8292ec3a6791482b847b5728af5c3012: stopping
tablet replica
I1207 02:40:47.920554 8717 raft_consensus.cc:2229] T
3dac2fd02f6045b3aa01d06e00d7fdf2 P 8292ec3a6791482b847b5728af5c3012 [term 1
LEADER]: Raft consensus shutting down.
I1207 02:40:47.920579 8717 raft_consensus.cc:2258] T
3dac2fd02f6045b3aa01d06e00d7fdf2 P 8292ec3a6791482b847b5728af5c3012 [term 1
FOLLOWER]: Raft consensus is shut down!
I1207 02:40:47.920653 8717 ts_tablet_manager.cc:1824] T
3dac2fd02f6045b3aa01d06e00d7fdf2 P 8292ec3a6791482b847b5728af5c3012: Deleting
tablet data with delete state TABLET_DATA_DELETED
I1207 02:40:47.921308 8717 ts_tablet_manager.cc:1837] T
3dac2fd02f6045b3aa01d06e00d7fdf2 P 8292ec3a6791482b847b5728af5c3012: tablet
deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I1207 02:40:47.921324 8717 log.cc:1192] T 3dac2fd02f6045b3aa01d06e00d7fdf2 P
8292ec3a6791482b847b5728af5c3012: Deleting WAL directory at
/home/ubuntu/Impala/testdata/cluster/cdh7/node-3/var/lib/kudu/ts/wal/wals/3dac2fd02f6045b3aa01d06e00d7fdf2
I1207 02:40:47.921389 8717 ts_tablet_manager.cc:1858] T
3dac2fd02f6045b3aa01d06e00d7fdf2 P 8292ec3a6791482b847b5728af5c3012: Deleting
consensus metadata
I1207 02:40:47.921643 8717 tablet_replica.cc:330] T
fa987644f7554383a1a9b6984e54cfa2 P 8292ec3a6791482b847b5728af5c3012: stopping
tablet replica
I1207 02:40:47.921677 8717 raft_consensus.cc:2229] T
fa987644f7554383a1a9b6984e54cfa2 P 8292ec3a6791482b847b5728af5c3012 [term 1
FOLLOWER]: Raft consensus shutting down.
I1207 02:40:47.921687 8717 raft_consensus.cc:2258] T
fa987644f7554383a1a9b6984e54cfa2 P 8292ec3a6791482b847b5728af5c3012 [term 1
FOLLOWER]: Raft consensus is shut down!
I1207 02:40:47.921737 8717 ts_tablet_manager.cc:1824] T
fa987644f7554383a1a9b6984e54cfa2 P 8292ec3a6791482b847b5728af5c3012: Deleting
tablet data with delete state TABLET_DATA_DELETED
I1207 02:40:47.922452 8717 ts_tablet_manager.cc:1837] T
fa987644f7554383a1a9b6984e54cfa2 P 8292ec3a6791482b847b5728af5c3012: tablet
deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I1207 02:40:47.922468 8717 log.cc:1192] T fa987644f7554383a1a9b6984e54cfa2 P
8292ec3a6791482b847b5728af5c3012: Deleting WAL directory at
/home/ubuntu/Impala/testdata/cluster/cdh7/node-3/var/lib/kudu/ts/wal/wals/fa987644f7554383a1a9b6984e54cfa2
I1207 02:40:47.922526 8717 ts_tablet_manager.cc:1858] T
fa987644f7554383a1a9b6984e54cfa2 P 8292ec3a6791482b847b5728af5c3012: Deleting
consensus metadata
I1207 02:40:47.922871 8717 tablet_replica.cc:330] T
b4318238b0e94b718881e5c6d0afd0e5 P 8292ec3a6791482b847b5728af5c3012: stopping
tablet replica
I1207 02:40:47.922904 8717 raft_consensus.cc:2229] T
b4318238b0e94b718881e5c6d0afd0e5 P 8292ec3a6791482b847b5728af5c3012 [term 1
LEADER]: Raft consensus shutting down.
I1207 02:40:47.922933 8717 raft_consensus.cc:2258] T
b4318238b0e94b718881e5c6d0afd0e5 P 8292ec3a6791482b847b5728af5c3012 [term 1
FOLLOWER]: Raft consensus is shut down!
I1207 02:40:47.923015 8717 ts_tablet_manager.cc:1824] T
b4318238b0e94b718881e5c6d0afd0e5 P 8292ec3a6791482b847b5728af5c3012: Deleting
tablet data with delete state TABLET_DATA_DELETED
I1207 02:40:47.923732 8717 ts_tablet_manager.cc:1837] T
b4318238b0e94b718881e5c6d0afd0e5 P 8292ec3a6791482b847b5728af5c3012: tablet
deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I1207 02:40:47.923758 8717 log.cc:1192] T b4318238b0e94b718881e5c6d0afd0e5 P
8292ec3a6791482b847b5728af5c3012: Deleting WAL directory at
/home/ubuntu/Impala/testdata/cluster/cdh7/node-3/var/lib/kudu/ts/wal/wals/b4318238b0e94b718881e5c6d0afd0e5
I1207 02:40:47.923826 8717 ts_tablet_manager.cc:1858] T
b4318238b0e94b718881e5c6d0afd0e5 P 8292ec3a6791482b847b5728af5c3012: Deleting
consensus metadata
I1207 02:40:47.924152 8717 tablet_replica.cc:330] T
daf8246510f14af1bab937551059d615 P 8292ec3a6791482b847b5728af5c3012: stopping
tablet replica
I1207 02:40:47.924182 8717 raft_consensus.cc:2229] T
daf8246510f14af1bab937551059d615 P 8292ec3a6791482b847b5728af5c3012 [term 1
LEADER]: Raft consensus shutting down.
I1207 02:40:47.924201 8717 raft_consensus.cc:2258] T
daf8246510f14af1bab937551059d615 P 8292ec3a6791482b847b5728af5c3012 [term 1
FOLLOWER]: Raft consensus is shut down!
I1207 02:40:47.924249 8717 ts_tablet_manager.cc:1824] T
daf8246510f14af1bab937551059d615 P 8292ec3a6791482b847b5728af5c3012: Deleting
tablet data with delete state TABLET_DATA_DELETED
I1207 02:40:47.924993 8717 ts_tablet_manager.cc:1837] T
daf8246510f14af1bab937551059d615 P 8292ec3a6791482b847b5728af5c3012: tablet
deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I1207 02:40:47.925007 8717 log.cc:1192] T daf8246510f14af1bab937551059d615 P
8292ec3a6791482b847b5728af5c3012: Deleting WAL directory at
/home/ubuntu/Impala/testdata/cluster/cdh7/node-3/var/lib/kudu/ts/wal/wals/daf8246510f14af1bab937551059d615
I1207 02:40:47.925066 8717 ts_tablet_manager.cc:1858] T
daf8246510f14af1bab937551059d615 P 8292ec3a6791482b847b5728af5c3012: Deleting
consensus metadata
{code}
Timing-wise, that looks like it's after all query attempts have completed and
the database is being cleaned up with {{DROP DATABASE
`test_read_modes_53f93f33` CASCADE}}.
> TestKuduOperations.test_read_modes fails
> ----------------------------------------
>
> Key: IMPALA-11187
> URL: https://issues.apache.org/jira/browse/IMPALA-11187
> Project: IMPALA
> Issue Type: Bug
> Reporter: Daniel Becker
> Assignee: Tamas Mate
> Priority: Minor
> Labels: broken-build
>
> In an internal build,
> *query_test.test_kudu.TestKuduOperations.test_read_modes* fails:
> h3. Error Message
> {code:java}
> query_test/test_kudu.py:551: in test_read_modes self._retry_query(cursor,
> "select count(*) from %s" % table_name, [(103,)])
> query_test/test_kudu.py:535: in _retry_query assert retries < 3, \ E
> AssertionError: Did not get a correct result for select count(*) from
> test_read_modes_53f93f33.test_read_latest after 3 retries: [(97,)] E assert 3
> < 3{code}
> h3. Stacktrace
> {code:java}
> query_test/test_kudu.py:551: in test_read_modes self._retry_query(cursor,
> "select count(*) from %s" % table_name, [(103,)])
> query_test/test_kudu.py:535: in _retry_query
> assert retries < 3, \
> E AssertionError: Did not get a correct result for select count(*) from
> test_read_modes_53f93f33.test_read_latest after 3 retries: [(97,)]
> E assert 3 < 3{code}
> h3. Standard Error
> {code:java}
> -- 2022-03-16 01:02:23,338 INFO MainThread: Using database
> testkuduoperations_1677_vu0h74 as default
> SET
> client_identifier=query_test/test_kudu.py::TestKuduOperations::()::test_read_modes;
> SET sync_ddl=False;
> -- executing against localhost:21000
> DROP DATABASE IF EXISTS `test_read_modes_53f93f33` CASCADE;
> -- 2022-03-16 01:02:23,342 INFO MainThread: Started query
> 2e4741a082f8195b:c8427d1b00000000
> SET
> client_identifier=query_test/test_kudu.py::TestKuduOperations::()::test_read_modes;
> SET sync_ddl=False;
> -- executing against localhost:21000
> CREATE DATABASE `test_read_modes_53f93f33`;
> -- 2022-03-16 01:02:28,252 INFO MainThread: Started query
> c547f10a0d52e296:62a357c600000000
> -- 2022-03-16 01:02:28,305 INFO MainThread: Created database
> "test_read_modes_53f93f33" for test ID
> "query_test/test_kudu.py::TestKuduOperations::()::test_read_modes"{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
