[ https://issues.apache.org/jira/browse/KUDU-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Serbin updated KUDU-1632: -------------------------------- Description: After switching tests from running in MANUAL_FLUSH mode into AUTO_FLUSH_BACKGROUND mode, the debug assert started to fire when running the all_types-itest built in debug configuration and KUDU_ALLOW_SLOW_TESTS=1. It's 100% reproducible: just run {noformat} KUDU_ALLOW_SLOW_TESTS=1 ./bin/all_types-itest 2>/tmp/all_types.log {noformat} The following stacktrace is reported by the test: {noformat} F0920 12:26:24.726744 13286 consensus_queue.cc:401] Check failed: request->ByteSize() <= FLAGS_consensus_max_batch_size_bytes (1168286 vs. 1048576) *** Check failure stack trace: *** @ 0x7f74c9f42294 google::LogMessage::SendToLog() @ 0x7f74c9f42790 google::LogMessage::Flush() @ 0x7f74c9f463c2 google::LogMessageFatal::~LogMessageFatal() @ 0x7f74cf109918 kudu::consensus::PeerMessageQueue::RequestForPeer() @ 0x7f74cf0fc302 kudu::consensus::Peer::SendNextRequest() @ 0x7f74cf0fdfaf kudu::consensus::Peer::DoProcessResponse() @ 0x7f74cf1050a9 kudu::internal::RunnableAdapter<>::Run() @ 0x7f74cf10502c kudu::internal::InvokeHelper<>::MakeItSo() @ 0x7f74cf104ffa kudu::internal::Invoker<>::Run() @ 0x7f74cf1284ce kudu::Callback<>::Run() @ 0x7f74cf12e7b9 boost::_mfi::cmf0<>::operator()() @ 0x7f74cf12e720 boost::_bi::list1<>::operator()<>() @ 0x7f74cf12e6ca boost::_bi::bind_t<>::operator()() @ 0x7f74cf12e430 boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x7f74ce164b98 boost::function0<>::operator()() @ 0x7f74cab47589 kudu::FunctionRunnable::Run() @ 0x7f74cab45994 kudu::ThreadPool::DispatchThread() @ 0x7f74cab49cc9 boost::_mfi::mf1<>::operator()() @ 0x7f74cab49c27 boost::_bi::list2<>::operator()<>() @ 0x7f74cab49baa boost::_bi::bind_t<>::operator()() @ 0x7f74cab49930 boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x7f74ce164b98 boost::function0<>::operator()() @ 0x7f74cab3ab30 kudu::Thread::SuperviseThread() @ 0x3ae0e079d1 (unknown) @ 0x3ae0ae88fd (unknown) @ (nil) (unknown) {noformat} The patch which introduces the change from MANUAL_FLUSH to AUTO_FLUSH_BACKGROUND can be found at: https://gerrit.cloudera.org/#/c/4471/ Also, if trying to reproduce the issue, set the {{consensus_max_batch_size_bytes}} to 1048576 (i.e. 1MiB): as a temporary workaround to avoid triggering the debug assert, the {{all_types-itest}} set it to 2MiB. was: After switching tests from running in MANUAL_FLUSH mode into AUTO_FLUSH_BACKGROUND mode, the debug assert started to fire when running the all_types-itest built in debug configuration and KUDU_ALLOW_SLOW_TESTS=1. It's 100% reproducible: just run {noformat} KUDU_ALLOW_SLOW_TESTS=1 ./bin/all_types-itest 2>/tmp/all_types.log {noformat} The following stacktrace is reported by the test: {noformat} F0920 12:26:24.726744 13286 consensus_queue.cc:401] Check failed: request->ByteSize() <= FLAGS_consensus_max_batch_size_bytes (1168286 vs. 1048576) *** Check failure stack trace: *** @ 0x7f74c9f42294 google::LogMessage::SendToLog() @ 0x7f74c9f42790 google::LogMessage::Flush() @ 0x7f74c9f463c2 google::LogMessageFatal::~LogMessageFatal() @ 0x7f74cf109918 kudu::consensus::PeerMessageQueue::RequestForPeer() @ 0x7f74cf0fc302 kudu::consensus::Peer::SendNextRequest() @ 0x7f74cf0fdfaf kudu::consensus::Peer::DoProcessResponse() @ 0x7f74cf1050a9 kudu::internal::RunnableAdapter<>::Run() @ 0x7f74cf10502c kudu::internal::InvokeHelper<>::MakeItSo() @ 0x7f74cf104ffa kudu::internal::Invoker<>::Run() @ 0x7f74cf1284ce kudu::Callback<>::Run() @ 0x7f74cf12e7b9 boost::_mfi::cmf0<>::operator()() @ 0x7f74cf12e720 boost::_bi::list1<>::operator()<>() @ 0x7f74cf12e6ca boost::_bi::bind_t<>::operator()() @ 0x7f74cf12e430 boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x7f74ce164b98 boost::function0<>::operator()() @ 0x7f74cab47589 kudu::FunctionRunnable::Run() @ 0x7f74cab45994 kudu::ThreadPool::DispatchThread() @ 0x7f74cab49cc9 boost::_mfi::mf1<>::operator()() @ 0x7f74cab49c27 boost::_bi::list2<>::operator()<>() @ 0x7f74cab49baa boost::_bi::bind_t<>::operator()() @ 0x7f74cab49930 boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x7f74ce164b98 boost::function0<>::operator()() @ 0x7f74cab3ab30 kudu::Thread::SuperviseThread() @ 0x3ae0e079d1 (unknown) @ 0x3ae0ae88fd (unknown) @ (nil) (unknown) {noformat} The patch which introduces the change from MANUAL_FLUSH to AUTO_FLUSH_BACKGROUND can be found at: https://gerrit.cloudera.org/#/c/4471/ > Assert on request size in consensus_queue.cc > -------------------------------------------- > > Key: KUDU-1632 > URL: https://issues.apache.org/jira/browse/KUDU-1632 > Project: Kudu > Issue Type: Bug > Components: consensus > Affects Versions: 1.0.0 > Environment: Development environment: Linux or MacOS X will do. > Build the project in debug configuration. > Reporter: Alexey Serbin > > After switching tests from running in MANUAL_FLUSH mode into > AUTO_FLUSH_BACKGROUND mode, the debug assert started to fire when running the > all_types-itest built in debug configuration and KUDU_ALLOW_SLOW_TESTS=1. > It's 100% reproducible: just run > {noformat} > KUDU_ALLOW_SLOW_TESTS=1 ./bin/all_types-itest 2>/tmp/all_types.log > {noformat} > The following stacktrace is reported by the test: > {noformat} > F0920 12:26:24.726744 13286 consensus_queue.cc:401] Check failed: > request->ByteSize() <= FLAGS_consensus_max_batch_size_bytes (1168286 vs. > 1048576) > *** Check failure stack trace: *** > @ 0x7f74c9f42294 google::LogMessage::SendToLog() > @ 0x7f74c9f42790 google::LogMessage::Flush() > @ 0x7f74c9f463c2 google::LogMessageFatal::~LogMessageFatal() > @ 0x7f74cf109918 kudu::consensus::PeerMessageQueue::RequestForPeer() > @ 0x7f74cf0fc302 kudu::consensus::Peer::SendNextRequest() > @ 0x7f74cf0fdfaf kudu::consensus::Peer::DoProcessResponse() > @ 0x7f74cf1050a9 kudu::internal::RunnableAdapter<>::Run() > @ 0x7f74cf10502c kudu::internal::InvokeHelper<>::MakeItSo() > @ 0x7f74cf104ffa kudu::internal::Invoker<>::Run() > @ 0x7f74cf1284ce kudu::Callback<>::Run() > @ 0x7f74cf12e7b9 boost::_mfi::cmf0<>::operator()() > @ 0x7f74cf12e720 boost::_bi::list1<>::operator()<>() > @ 0x7f74cf12e6ca boost::_bi::bind_t<>::operator()() > @ 0x7f74cf12e430 > boost::detail::function::void_function_obj_invoker0<>::invoke() > @ 0x7f74ce164b98 boost::function0<>::operator()() > @ 0x7f74cab47589 kudu::FunctionRunnable::Run() > @ 0x7f74cab45994 kudu::ThreadPool::DispatchThread() > @ 0x7f74cab49cc9 boost::_mfi::mf1<>::operator()() > @ 0x7f74cab49c27 boost::_bi::list2<>::operator()<>() > @ 0x7f74cab49baa boost::_bi::bind_t<>::operator()() > @ 0x7f74cab49930 > boost::detail::function::void_function_obj_invoker0<>::invoke() > @ 0x7f74ce164b98 boost::function0<>::operator()() > @ 0x7f74cab3ab30 kudu::Thread::SuperviseThread() > @ 0x3ae0e079d1 (unknown) > @ 0x3ae0ae88fd (unknown) > @ (nil) (unknown) > {noformat} > The patch which introduces the change from MANUAL_FLUSH to > AUTO_FLUSH_BACKGROUND can be found at: https://gerrit.cloudera.org/#/c/4471/ > Also, if trying to reproduce the issue, set the > {{consensus_max_batch_size_bytes}} to 1048576 (i.e. 1MiB): as a temporary > workaround to avoid triggering the debug assert, the {{all_types-itest}} set > it to 2MiB. -- This message was sent by Atlassian JIRA (v6.3.4#6332)