[
https://issues.apache.org/jira/browse/IMPALA-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785348#comment-17785348
]
Wenzhe Zhou commented on IMPALA-12556:
--------------------------------------
Thanks [~laszlog] to file this Jira. I will work on it.
Catalogd log messages.
{code:java}
F1111 23:36:24.798915 2270244 statestore-subscriber.cc:336] Check failed:
!statestore_is_active || !statestore2_is_active
*** Check failure stack trace: ***
@ 0x389f66d google::LogMessage::Fail()
@ 0x38a15a4 google::LogMessage::SendToLog()
@ 0x389f04c google::LogMessage::Flush()
I1111 23:36:24.834599 2270278 statestore-subscriber.cc:743] Starting statestore
subscriber
I1111 23:36:24.835072 2272522 TAcceptQueueServer.cpp:355] New connection to
server StatestoreSubscriber from client <Host: 127.0.0.1 Port: 48662>
I1111 23:36:24.835194 2272522 TAcceptQueueServer.cpp:355] New connection to
server StatestoreSubscriber from client <Host: 127.0.0.1 Port: 48668>
I1111 23:36:24.835242 2272522 TAcceptQueueServer.cpp:355] New connection to
server StatestoreSubscriber from client <Host: 127.0.0.1 Port: 48674>
I1111 23:36:24.835269 2270278 statestore-subscriber.cc:684] Statestore protocol
version: V2
I1111 23:36:24.835278 2270278 statestore-subscriber.cc:693] Subscriber
registration ID: b7427b8c816720ad:597a82d4368d7792
I1111 23:36:24.835280 2270278 statestore-subscriber.cc:701] Statestore ID:
4b4c5e6402fee001:ddd1f305755eb79f
I1111 23:36:24.835281 2270278 statestore-subscriber.cc:719] Active catalogd
address: ip-172-31-52-128:26000
I1111 23:36:24.835283 2270278 statestore-subscriber.cc:753] statestore
registration successful on startup
I1111 23:36:24.835287 2270278 exec-env.cc:715] The address of Catalog service
is changed from localhost:26000 to ip-172-31-52-128:26000
I1111 23:36:24.835299 2272522 TAcceptQueueServer.cpp:355] New connection to
server StatestoreSubscriber from client <Host: 127.0.0.1 Port: 48676>
I1111 23:36:24.835448 2270278 statestore-subscriber.cc:743] Starting statestore
subscriber
I1111 23:36:24.835820 2270278 statestore-subscriber.cc:684] Statestore protocol
version: V2
I1111 23:36:24.835829 2270278 statestore-subscriber.cc:693] Subscriber
registration ID: ae4e3d7426091e84:2d5e2ba69055ca89
I1111 23:36:24.835831 2270278 statestore-subscriber.cc:701] Statestore ID:
6d493a7087e09773:97eb4a1e889718b7
I1111 23:36:24.835834 2270278 statestore-subscriber.cc:719] Active catalogd
address: ip-172-31-52-128:26000
I1111 23:36:24.835836 2270278 statestore-subscriber.cc:753] statestore
registration successful on startup
I1111 23:36:24.835909 2272522 TAcceptQueueServer.cpp:355] New connection to
server StatestoreSubscriber from client <Host: 127.0.0.1 Port: 48682>
F1111 23:36:24.835925 2270278 statestore-subscriber.cc:336] Check failed:
!statestore_is_active || !statestore2_is_active
@ 0x38a1ac9 google::LogMessageFatal::~LogMessageFatal()
@ 0x1811a13 impala::StatestoreSubscriber::Start()
@ 0xfb419a impala::CatalogServer::Start()
@ 0xeffb37 CatalogdMain()
@ 0xefe4bf main
@ 0x7f2a2c2ec083 __libc_start_main
@ 0xefe2fe _start
Minidump with no thread info available.
{code}
impalad log messages:
{code:java}
F1111 23:36:24.835925 2270278 statestore-subscriber.cc:336] Check failed:
!statestore_is_active || !statestore2_is_active
*** Check failure stack trace: ***
@ 0x389f66d google::LogMessage::Fail()
@ 0x38a15a4 google::LogMessage::SendToLog()
@ 0x389f04c google::LogMessage::Flush()
@ 0x38a1ac9 google::LogMessageFatal::~LogMessageFatal()
@ 0x1811a13 impala::StatestoreSubscriber::Start()
@ 0x1391180 impala::ExecEnv::StartStatestoreSubscriberService()
@ 0x1625138 impala::ImpalaServer::Start()
@ 0x1609175 ImpaladMain()
@ 0xefe49c main
@ 0x7f77fc990083 __libc_start_main
@ 0xefe2fe _start
Minidump with no thread info available.
{code}
statestored log messages:
{code:java}
I1111 23:36:22.327819 2270287 TAcceptQueueServer.cpp:355] New connection to
server StatestoreHaService from client <Host: 127.0.0.1 Port: 36042>
I1111 23:36:22.327908 2270228 statestore.cc:1878] Send Statestore HA handshake
request
I1111 23:36:22.327960 2270287 TAcceptQueueServer.cpp:355] New connection to
server StatestoreHaService from client <Host: 127.0.0.1 Port: 36050>
I1111 23:36:22.328279 2270299 statestore.cc:1910] Receive Statestore HA
handshake request
I1111 23:36:22.328285 2270299 statestore.cc:1917] Set the statestored as
standby since the peer is started with force active flag
I1111 23:36:22.328310 2270228 statestore.cc:1851] Receive Statestore HA
handshake response, set the statestore as active
I1111 23:36:22.328430 2270302 TAcceptQueueServer.cpp:329]
connection_setup_thread_pool_size is set to 2
I1111 23:36:22.328433 2270228 thrift-server.cc:419] ThriftServer
'StatestoreService' started on port: 24001
I1111 23:36:22.427978 2270296 statestore.cc:1640] Send notification for active
statestored version: 1699745782328309
I1111 23:36:22.427994 2270296 statestore.cc:1714] Successfully sent
UpdateStatestoredRole RPCs to all subscribers
{code}
> test_two_statestored_with_force_active fails or flaky
> -----------------------------------------------------
>
> Key: IMPALA-12556
> URL: https://issues.apache.org/jira/browse/IMPALA-12556
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
> Affects Versions: Impala 4.4.0
> Reporter: Laszlo Gaal
> Assignee: Wenzhe Zhou
> Priority: Blocker
>
> custom_cluster.test_statestored_ha.TestStatestoredHA.test_two_statestored_with_force_active
> failed in a precommit run.
> Symptom:
> {code}
> common/custom_cluster_test_suite.py:208: in setup_method
> self._start_impala_cluster(cluster_args, **kwargs)
> common/custom_cluster_test_suite.py:330: in _start_impala_cluster
> check_call(cmd + options, close_fds=True)
> ../toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/subprocess.py:190:
> in check_call
> raise CalledProcessError(retcode, cmd)
> E CalledProcessError: Command
> '['/home/ubuntu/Impala/bin/start-impala-cluster.py',
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50', '--cluster_size=3',
> '--num_coordinators=3',
> '--log_dir=/home/ubuntu/Impala/logs/custom_cluster_tests', '--log_level=1',
> '--state_store_args=--statestore_force_active=true ',
> '--enable_statestored_ha', '--impalad_args=--default_query_options=']'
> returned non-zero exit status 1
> {code}
> The test dies with a FATAL log entry in catalogd's log:
> {code}
> DCHECK found in log file:
> /home/ubuntu/Impala/logs/custom_cluster_tests/catalogd.FATAL
> {code}
> {code}
> Log file created at: 2023/11/11 23:36:24
> Running on machine: ip-172-31-52-128
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> F1111 23:36:24.798915 2270244 statestore-subscriber.cc:336] Check failed:
> !statestore_is_active || !statestore2_is_active
> {code}
> Offending precommit run:
> https://jenkins.impala.io/job/ubuntu-20.04-from-scratch/874/ (preserved).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
