[
https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847942#comment-16847942
]
Sahil Takiar commented on IMPALA-8577:
--------------------------------------
I can re-produce a similar error locally during an ASAN build (unfortunately
the Java stack is missing):
{code:java}
==61922==ERROR: AddressSanitizer: attempting double-free on 0x613000561280 in
thread T179:
#0 0x17e7c30 in free
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_m
alloc_linux.cc:47
#1 0x7f58a642a0cc in CRYPTO_free
(/lib/x86_64-linux-gnu/libcrypto.so.1.0.0+0x630cc)
#2 0x7f58a684f02e (/lib/x86_64-linux-gnu/libssl.so.1.0.0+0x4302e)
#3 0x7f58a6827add (/lib/x86_64-linux-gnu/libssl.so.1.0.0+0x1badd)
#4 0x7f58a682d55e (/lib/x86_64-linux-gnu/libssl.so.1.0.0+0x2155e)
#5 0x7f542977aab1 in Java_org_wildfly_openssl_SSLImpl_doHandshake0
(/tmp/tmp-6414427375012512006openssl/libwfssl.so+0x9ab1)
#6 0x7f589038ad90 (<unknown module>)
0x613000561280 is located 0 bytes inside of 352-byte region
[0x613000561280,0x6130005613e0)
freed by thread T187 here:
==61922==AddressSanitizer CHECK failed:
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_descriptions.cc:176
"((id)) != (0)" (0x0, 0x0)
#0 0x17f2a2f in __asan::AsanCheckFailed(char const*, int, char const*, unsigned
long long, unsigned long long)
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_rtl.cc:69
#1 0x180e705 in __sanitizer::CheckFailed(char const*, int, char const*,
unsigned long long, unsigned long long)
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/sanitizer_common/sanitizer_termination.cc:79
#2 0x1723be4 in GetStackTraceFromId
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_descriptions.cc:176
#3 0x1723be4 in __asan::HeapAddressDescription::Print() const
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_descriptions.cc:398
#4 0x1725227 in __asan::ErrorDoubleFree::Print()
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_errors.cc:117
#5 0x17eed0b in __asan::ErrorDescription::Print()
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_errors.h:374
#6 0x17eed0b in ~ScopedInErrorReport
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_report.cc:177
#7 0x17eed0b in __asan::ReportDoubleFree(unsigned long,
__sanitizer::BufferedStackTrace*)
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_report.cc:279
#8 0x171fe3a in __asan::Allocator::ReportInvalidFree(void*, unsigned char,
__sanitizer::BufferedStackTrace*)
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_allocator.cc:652
#9 0x171fe3a in
__asan::Allocator::AtomicallySetQuarantineFlagIfAllocated(__asan::AsanChunk*,
void*, __sanitizer::BufferedStackTrace*)
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_allocator.cc:523
#10 0x171fe3a in __asan::Allocator::Deallocate(void*, unsigned long,
__sanitizer::BufferedStackTrace*, __asan::AllocType)
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_allocator.cc:597
#11 0x171fe3a in __asan::asan_free(void*, __sanitizer::BufferedStackTrace*,
__asan::AllocType)
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_allocator.cc:805
#12 0x17e7c0c in free
/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:50
#13 0x7f58a642a0cc in CRYPTO_free
(/lib/x86_64-linux-gnu/libcrypto.so.1.0.0+0x630cc)
#14 0x7f58a684f02e (/lib/x86_64-linux-gnu/libssl.so.1.0.0+0x4302e)
#15 0x7f58a6827add (/lib/x86_64-linux-gnu/libssl.so.1.0.0+0x1badd)
#16 0x7f58a682d55e (/lib/x86_64-linux-gnu/libssl.so.1.0.0+0x2155e)
#17 0x7f542977aab1 in Java_org_wildfly_openssl_SSLImpl_doHandshake0
(/tmp/tmp-6414427375012512006openssl/libwfssl.so+0x9ab1)
#18 0x7f589038ad90 (<unknown module>){code}
Still need to figure out why this is happening.
> Crash during OpenSSLSocket.read
> -------------------------------
>
> Key: IMPALA-8577
> URL: https://issues.apache.org/jira/browse/IMPALA-8577
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.3.0
> Reporter: David Rorke
> Assignee: Sahil Takiar
> Priority: Major
> Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp,
> hs_err_pid6313.log,
> impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313,
>
> impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313
>
>
> Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from
> the stack trace (hs_err log file attached with more complete stack):
> {noformat}
> Stack: [0x00007f3d095bc000,0x00007f3d09dbc000], sp=0x00007f3d09db9050, free
> space=8180k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C [impalad+0x2528a33]
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
> unsigned long, int)+0x133
> C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f
> C [impalad+0x266468a] operator delete(void*)+0x32a
> C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d
> J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @
> 0x00007f3d4dadf9f9 [0x00007f3d4dadf940+0xb9]
> J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @
> 0x00007f3d4dfd0dfc [0x00007f3d4dfd0d80+0x7c]
> J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @
> 0x00007f3d4de4fe2c [0x00007f3d4de4f720+0x70c]
> J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @
> 0x00007f3d4de419cc [0x00007f3d4de417c0+0x20c]
> J 2994 C2
> org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult;
> (892 bytes) @ 0x00007f3d4db8da34 [0x00007f3d4db8c900+0x1134]
> J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @
> 0x00007f3d4dd64cb0 [0x00007f3d4dd646c0+0x5f0]
> J 5090 C2
> com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I
> (97 bytes) @ 0x00007f3d4ddd9ee0 [0x00007f3d4ddd9e40+0xa0]
> J 5846 C1
> com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I
> (48 bytes) @ 0x00007f3d4d7acb24 [0x00007f3d4d7ac7a0+0x384]
> J 5845 C1
> com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31
> bytes) @ 0x00007f3d4d7ad49c [0x00007f3d4d7ad220+0x27c]
> {noformat}
> The crash may not be easy to reproduce. I've run this test multiple times
> and only crashed once. I have a core file if needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
