hust-hhb opened a new pull request, #41174: URL: https://github.com/apache/doris/pull/41174
now be may be core when allocatoring memory fail on load_pk_index_and_bf SLF4J: Found binding in [jar:file:/mnt/disk1/CloudEnv1Stress/cluster0/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] start BE in cloud mode, cloud_unique_id: 1:CloudStressEnv1Cluster0:CLOUD_STRESS_ENV1_COMPUTE_ID, meta_service_endpoint: 172.20.48.42:5000 thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift e rror, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, re ason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THR IFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=thrift erro r, reason=thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed ou t)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift err or, reason=THRIFT_EAGAIN (timed out)THRIFT_EAGAIN (timed out)THRIFT_EAGAIN (timed out)THRIFT_EAGAIN (timed out)THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=thrift error, reason=THRIFT_EAGAIN (timed out)THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)terminate called a fter throwing an instance of 'doris::Exception' what(): [E11] Allocator sys memory check failed: Cannot alloc:596, consuming tracker:<PKIndexPageCache[size](AllocByAllocator)>, peak used 5238306191, current used 5235962558, exec node:<>, process memory used 49.89 GB exceed limit 49. 13 GB or sys available memory 10.80 GB less than low water mark 3.07 GB. 0# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/exception.cpp:0 1# Allocator<false, false, false, DefaultMemoryAllocator>::sys_memory_check(unsigned long) const at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/common/allocator.cpp:159 2# Allocator<false, false, false, DefaultMemoryAllocator>::alloc_impl(unsigned long, unsigned long) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/common/allocator.cpp:207 3# doris::PageBase<Allocator<false, false, false, DefaultMemoryAllocator> >::PageBase(unsigned long, bool, doris::segment_v2::PageTypePB) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/page_cache.cpp:37 4# doris::segment_v2::PageIO::read_and_decompress_page(doris::segment_v2::PageReadOptions const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64- linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:155 5# doris::segment_v2::IndexedColumnReader::read_page(doris::segment_v2::PagePointer const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*, doris::segment_v2::PageTypePB, doris::BlockCompressionCo dec*, bool) const at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/indexed_column_reader.cpp:134 6# doris::segment_v2::IndexedColumnReader::load_index_page(doris::segment_v2::PagePointerPB const&, doris::segment_v2::PageHandle*, doris::segment_v2::IndexPageReader*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/ rowset/segment_v2/indexed_column_reader.cpp:107 7# doris::segment_v2::IndexedColumnReader::load(bool, bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/status.h:498 8# doris::PrimaryKeyIndexReader::parse_index(std::shared_ptr<doris::io::FileReader>, doris::segment_v2::PrimaryKeyIndexMetaPB const&) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/status.h:498 9# doris::segment_v2::Segment::_load_index_impl() at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701 10# doris::segment_v2::Segment::new_iterator(std::shared_ptr<doris::Schema const>, doris::StorageReadOptions const&, std::unique_ptr<doris::RowwiseIterator, std::default_delete<doris::RowwiseIterator> >*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/status.h:498 11# doris::BetaRowsetReader::get_segment_iterators(doris::RowsetReaderContext*, std::vector<std::unique_ptr<doris::RowwiseIterator, std::default_delete<doris::RowwiseIterator> >, std::allocator<std::unique_ptr<doris::RowwiseIterat or, std::default_delete<doris::RowwiseIterator> > > >*, bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/status.h:387 12# doris::vectorized::VerticalBlockReader::_get_segment_iterators(doris::TabletReader::ReaderParams const&, std::vector<std::unique_ptr<doris::RowwiseIterator, std::default_delete<doris::RowwiseIterator> >, std::allocator<std::un ique_ptr<doris::RowwiseIterator, std::default_delete<doris::RowwiseIterator> > > >*, std::vector<bool, std::allocator<bool> >*, std::vector<doris::RowsetId, std::allocator<doris::RowsetId> >*) at /home/zcp/repo_center/doris_branch-3.0/dor is/be/src/vec/olap/vertical_block_reader.cpp:0 13# doris::vectorized::VerticalBlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/status.h:498 14# doris::vectorized::VerticalBlockReader::init(doris::TabletReader::ReaderParams const&, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/status.h:498 15# doris::Merger::vertical_compact_one_group(std::shared_ptr<doris::BaseTablet>, doris::ReaderType, doris::TabletSchema const&, bool, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::RowSourcesB uffer*, std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector<unsigned int, std::allocator<unsigned int> >, lo ng, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/merger.cpp:0 16# doris::Merger::vertical_merge_rowsets(std::shared_ptr<doris::BaseTablet>, doris::ReaderType, doris::TabletSchema const&, std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/merger.cpp:475 17# doris::Compaction::merge_input_rowsets() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/compaction.cpp:188 18# doris::CloudCompactionMixin::execute_compact_impl(long) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/status.h:498 19# doris::CloudCompactionMixin::execute_compact() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/status.h:498 20# doris::CloudCumulativeCompaction::execute_compact() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/status.h:498 21# std::_Function_handler<void (), doris::CloudStorageEngine::_submit_cumulative_compaction_task(std::shared_ptr<doris::CloudTablet> const&)::$_1>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_branch-3.0/doris/ be/src/common/status.h:498 22# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/threadpool.cpp:0 23# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562 24# ? 25# ? *** Query id: 5341c0fd4584ca6d-6a060d01d8eefad *** *** is nereids: 0 *** *** tablet id: 0 *** *** Aborted at 1727096053 (unix time) try "date -d @1727096053" if you are using GNU date *** *** Current BE git commitID: ebb7b64624 *** *** SIGABRT unknown detail explain (@0x27f016) received by PID 2617366 (TID 2694001 OR 0x7f37609fc640) from PID 2617366; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/signal_handler.h:421 1# 0x00007F5036778520 in /lib/x86_64-linux-gnu/libc.so.6 2# pthread_kill at ./nptl/pthread_kill.c:89 3# raise at ../sysdeps/posix/raise.c:27 4# abort at ./stdlib/abort.c:81 5# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75 6# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 7# 0x0000556C98D39801 in /mnt/disk1/CloudEnv1Stress/cluster0/be/lib/doris_be 8# __cxxabiv1::__terminate(void (*)()) [clone .cold] in /mnt/disk1/CloudEnv1Stress/cluster0/be/lib/doris_be 9# doris::segment_v2::Segment::_load_index_impl() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment.cpp:405 10# doris::segment_v2::Segment::load_pk_index_and_bf() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/rowset/segment_v2/segment.cpp:391 11# doris::segment_v2::Segment::lookup_row_key(doris::Slice const&, doris::TabletSchema const*, bool, bool, doris::RowLocation*) in /mnt/disk1/CloudEnv1Stress/cluster0/be/lib/doris_be 12# doris::BaseTablet::lookup_row_key(doris::Slice const&, doris::TabletSchema*, bool, std::vector<std::shared_ptr<doris::Rowset>, std::allocator<std::shared_ptr<doris::Rowset> > > const&, doris::RowLocation*, unsigned int, std::vector<st d::unique_ptr<doris::SegmentCacheHandle, std::default_delete<doris::SegmentCacheHandle> >, std::allocator<std::unique_ptr<doris::SegmentCacheHandle, std::default_delete<doris::SegmentCacheHandle> > > >&, std::shared_ptr<doris::Rowset>*, b ool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/base_tablet.cpp:532 13# doris::BaseTablet::calc_segment_delete_bitmap(std::shared_ptr<doris::Rowset>, std::shared_ptr<doris::segment_v2::Segment> const&, std::vector<std::shared_ptr<doris::Rowset>, std::allocator<std::shared_ptr<doris::Rowset> > > const&, st d::shared_ptr<doris::DeleteBitmap>, long, doris::RowsetWriter*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/olap/base_tablet.cpp:723 14# std::_Function_handler<void (), doris::CalcDeleteBitmapToken::submit(std::shared_ptr<doris::BaseTablet>, std::shared_ptr<doris::Rowset>, std::shared_ptr<doris::segment_v2::Segment> const&, std::vector<std::shared_ptr<doris::Rowset>, s td::allocator<std::shared_ptr<doris::Rowset> > > const&, long, std::shared_ptr<doris::DeleteBitmap>, doris::RowsetWriter*)::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../ include/c++/11/bits/std_function.h:291 15# doris::ThreadPool::dispatch_thread() in /mnt/disk1/CloudEnv1Stress/cluster0/be/lib/doris_be 16# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/thread.cpp:499 17# start_thread at ./nptl/pthread_create.c:442 18# 0x00007F503685C850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
