[
https://issues.apache.org/jira/browse/IMPALA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-8270 started by Tim Armstrong.
---------------------------------------------
> ASAN issue with MemTracker::LogUsage() called via webserver's /memz page
> ------------------------------------------------------------------------
>
> Key: IMPALA-8270
> URL: https://issues.apache.org/jira/browse/IMPALA-8270
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.2.0
> Reporter: Joe McDonnell
> Assignee: Tim Armstrong
> Priority: Blocker
> Labels: broken-build
>
> I saw this on an ASAN build from a several days ago:
> {noformat}
> ==124622==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x61200337f2d8 at pc 0x000001fdbdc5 bp 0x7f3b9e11db90 sp 0x7f3b9e11db88
> READ of size 8 at 0x61200337f2d8 thread T145092 (sq_worker)
> #0 0x1fdbdc4 in impala::MemTracker::LogUsage(int, std::string const&, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-tracker.cc:297:7
> #1 0x1fded9a in impala::MemTracker::LogUsage(int, std::string const&,
> std::list<impala::MemTracker*, std::allocator<impala::MemTracker*> > const&,
> long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-tracker.cc:362:36
> #2 0x1fdbb6c in impala::MemTracker::LogUsage(int, std::string const&, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-tracker.cc:338:28
> #3 0x1fded9a in impala::MemTracker::LogUsage(int, std::string const&,
> std::list<impala::MemTracker*, std::allocator<impala::MemTracker*> > const&,
> long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-tracker.cc:362:36
> #4 0x1fdbb6c in impala::MemTracker::LogUsage(int, std::string const&, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-tracker.cc:338:28
> #5 0x1fded9a in impala::MemTracker::LogUsage(int, std::string const&,
> std::list<impala::MemTracker*, std::allocator<impala::MemTracker*> > const&,
> long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-tracker.cc:362:36
> #6 0x1fdbb6c in impala::MemTracker::LogUsage(int, std::string const&, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-tracker.cc:338:28
> #7 0x241766f in MemUsageHandler(impala::MemTracker*, impala::MetricGroup*,
> std::map<std::string, std::string, std::less<std::string>,
> std::allocator<std::pair<std::string const, std::string> > > const&,
> rapidjson::GenericDocument<rapidjson::UTF8<char>,
> rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>,
> rapidjson::CrtAllocator>*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/default-path-handlers.cc:155:31
> #8 0x25296e5 in boost::function2<void, std::map<std::string, std::string,
> std::less<std::string>, std::allocator<std::pair<std::string const,
> std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>,
> rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>,
> rapidjson::CrtAllocator>*>::operator()(std::map<std::string, std::string,
> std::less<std::string>, std::allocator<std::pair<std::string const,
> std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>,
> rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>,
> rapidjson::CrtAllocator>*) const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #9 0x2527849 in
> impala::Webserver::RenderUrlWithTemplate(std::map<std::string, std::string,
> std::less<std::string>, std::allocator<std::pair<std::string const,
> std::string> > > const&, impala::Webserver::UrlHandler const&,
> std::basic_stringstream<char, std::char_traits<char>, std::allocator<char>
> >*, impala::ContentType*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/webserver.cc:447:3
> #10 0x2526ea7 in impala::Webserver::BeginRequestCallback(sq_connection*,
> sq_request_info*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/webserver.cc:419:5
> #11 0x253f1bf in handle_request
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x253f1bf)
> #12 0x253ebed in process_new_connection
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x253ebed)
> #13 0x253e616 in worker_thread
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x253e616)
> #14 0x7f45dfbdbe24 in start_thread (/lib64/libpthread.so.0+0x7e24)
> #15 0x7f45df6f234c in __clone (/lib64/libc.so.6+0xf834c)
> 0x61200337f2d8 is located 152 bytes inside of 312-byte region
> [0x61200337f240,0x61200337f378)
> freed by thread T138428 here:
> #0 0x17ce6c0 in operator delete(void*)
> /mnt/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:137
> #1 0x200ed3c in impala::RuntimeState::~RuntimeState()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/runtime-state.cc:111:1
> #2 0x2212e86 in
> Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/fe-support.cc:275:1
> #3 0x7f45c60d1d74 (<unknown module>)
> #4 0x7f45c6e0921b (<unknown module>)
> previously allocated by thread T138428 here:
> #0 0x17cd948 in operator new(unsigned long)
> /mnt/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:92
> #1 0x200dc63 in impala::RuntimeState::Init()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/runtime-state.cc:129:31
> #2 0x200e83e in impala::RuntimeState::RuntimeState(impala::TQueryCtx const&,
> impala::ExecEnv*, impala::DescriptorTbl*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/runtime-state.cc:106:3
> #3 0x2212d33 in
> Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/fe-support.cc:193:16
> #4 0x7f45c60d1d74 (<unknown module>)
> #5 0x7f45c6e0921b (<unknown module>)
> Thread T145092 (sq_worker) created by T273 (sq_acceptor) here:
> #0 0x16de83d in __interceptor_pthread_create
> /mnt/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x2535fd4 in sq_start_thread
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x2535fd4)
> #2 0x253e42d in try_start_another_worker
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x253e42d)
> #3 0x253e2bf in produce_socket
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x253e2bf)
> #4 0x253e11b in accept_new_connection
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x253e11b)
> #5 0x253b300 in master_thread
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x253b300)
> #6 0x7f45dfbdbe24 in start_thread (/lib64/libpthread.so.0+0x7e24)
> Thread T273 (sq_acceptor) created by T0 here:
> #0 0x16de83d in __interceptor_pthread_create
> /mnt/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x2535fd4 in sq_start_thread
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x2535fd4)
> #2 0x253a5db in sq_start
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x253a5db)
> #3 0x2525504 in impala::Webserver::Start()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/webserver.cc:318:14
> #4 0x1f6ba7a in impala::ExecEnv::Init()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/exec-env.cc:334:5
> #5 0x22274e0 in ImpaladMain(int, char**)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/impalad-main.cc:72:3
> #6 0x17d1d7d in main
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/daemon-main.cc:37:12
> #7 0x7f45df61bc04 in __libc_start_main (/lib64/libc.so.6+0x21c04)
> Thread T138428 created by T303 here:
> #0 0x16de83d in __interceptor_pthread_create
> /mnt/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x3cd4b69 in boost::thread::start_thread_noexcept()
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x3cd4b69)
> #2 0x45e0360d (<unknown module>)
> Thread T303 created by T301 here:
> #0 0x16de83d in __interceptor_pthread_create
> /mnt/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x3cd4b69 in boost::thread::start_thread_noexcept()
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x3cd4b69)
> #2 0x45e0360d (<unknown module>)
> Thread T301 created by T0 here:
> #0 0x16de83d in __interceptor_pthread_create
> /mnt/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
> #1 0x3cd4b69 in boost::thread::start_thread_noexcept()
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x3cd4b69)
> #2 0x45e0360d (<unknown module>)
> SUMMARY: AddressSanitizer: heap-use-after-free
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-tracker.cc:297:7
> in impala::MemTracker::LogUsage(int, std::string const&, long*)
> Shadow bytes around the buggy address:
> 0x0c2480667e00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c2480667e10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c2480667e20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c2480667e30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c2480667e40: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
> =>0x0c2480667e50: fd fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd
> 0x0c2480667e60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
> 0x0c2480667e70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c2480667e80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c2480667e90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c2480667ea0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Container overflow: fc
> Array cookie: ac
> Intra object redzone: bb
> ASan internal: fe
> Left alloca redzone: ca
> Right alloca redzone: cb
> ==124622==ABORTING{noformat}
> It looks like /memz's MemUsageHandler() is walking through MemTrackers, but
> accesses some freed memory from FeSupport. The webserver/test_web_pages.py
> tests run in parallel with other things, so that might create the conditions
> for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
