[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034648#comment-17034648 ] Sahil Takiar commented on IMPALA-8577: -- Can't remember for sure, but I think it was 1.0.2 > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034609#comment-17034609 ] Steve Loughran commented on IMPALA-8577: which openssl version? 1.0.4 doesn't seem to work with openssl 1.1.1 *at all*, forcing some backport fun > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009910#comment-17009910 ] Sahil Takiar commented on IMPALA-8577: -- Doesn't seem like this is specific to the AWS SDK, I just reproduced this on an older version of Widfly (1.0.4.Final), while running Impala-on-ABFS. {code:java} C [impalad+0x4bba003] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int)+0x133 C [impalad+0x4bba09c] tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long)+0x1c C [impalad+0x4cf4a80] operator delete(void*)+0x3c0 C [libcrypto.so.1.0.0+0x630cd] CRYPTO_free+0x1d J 4308 org.wildfly.openssl.SSLImpl.freeSSL0(J)V (0 bytes) @ 0x7f107ccdfeb1 [0x7f107ccdfe00+0xb1] J 4307 C1 org.wildfly.openssl.SSLImpl.freeSSL(J)V (5 bytes) @ 0x7f107ccdf404 [0x7f107ccdf380+0x84] J 4182 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (67 bytes) @ 0x7f107cc0384c [0x7f107cc03380+0x4cc] J 4321 C1 org.wildfly.openssl.OpenSSLSocket.close()V (78 bytes) @ 0x7f107cce6a9c [0x7f107cce65e0+0x4bc] J 3880 C1 sun.net.www.http.ClientVector.put(Lsun/net/www/http/HttpClient;)V (34 bytes) @ 0x7f107ca97404 [0x7f107ca96f60+0x4a4] j sun.net.www.http.KeepAliveCache.put(Ljava/net/URL;Ljava/lang/Object;Lsun/net/www/http/HttpClient;)V+138 J 4634 C2 sun.net.www.protocol.https.HttpsClient.putInKeepAliveCache()V (45 bytes) @ 0x7f107ce7e550 [0x7f107ce7e4e0+0x70] J 4134 C2 sun.net.www.protocol.http.HttpURLConnection.getInputStream0()Ljava/io/InputStream; (2023 bytes) @ 0x7f107cc25bc0 [0x7f107cc23000+0x2bc0] J 4215 C2 java.net.HttpURLConnection.getResponseCode()I (152 bytes) @ 0x7f107cc918d4 [0x7f107cc91800+0xd4] J 4447 C2 org.apache.hadoop.fs.azurebfs.services.AbfsHttpOperation.processResponse([BII)V (552 bytes) @ 0x7f107cd6c7f0 [0x7f107cd6c760+0x90] J 4577 C2 org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(I)Z (424 bytes) @ 0x7f107ceb3ed0 [0x7f107ceb3060+0xe70] J 4642 C2 org.apache.hadoop.fs.azurebfs.services.AbfsClient.getPathStatus(Ljava/lang/String;)Lorg/apache/hadoop/fs/azurebfs/services/AbfsRestOperation; (63 bytes) @ 0x7f107cf2ab4c [0x7f107cf2a0e0+0xa6c] J 4648 C2 org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.open(Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream; (75 bytes) @ 0x7f107cf78f20 [0x7f107cf77800+0x1720] {code} > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997434#comment-16997434 ] Sahil Takiar commented on IMPALA-8577: -- The new WildFly release + AWS SDK release seemed to do the trick, with the updated versions I can no longer reproduce this bug in Impala. > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874572#comment-16874572 ] Sahil Takiar commented on IMPALA-8577: -- I see a new wildfly openssl release: [https://github.com/wildfly/wildfly-openssl/releases/tag/1.0.7.Final] and looks like the AWS SDK has been updated. Should re-test with the upgraded versions. > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856717#comment-16856717 ] Steve Loughran commented on IMPALA-8577: I've reverted the entire feature: too many problems right now. Sorry. It's just all the issues were piling up on me. There's a new SDK update going in HADOOP-16117; maybe you can try again with that. > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853865#comment-16853865 ] Sahil Takiar commented on IMPALA-8577: -- I still can't reproduce the {{SSL_set_session}} bug, I noticed some issues with my environment that I'm trying to fix, so maybe thats what caused it. wildly-openssl is suppose to work as a drop in replacement for JSSE, although there is the possibility that the bug *does* exist in the AWS SDK and the JSSE implementation is just more defensive. Am out of town for a bit, but will re-visit this in about two weeks. > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853338#comment-16853338 ] Sahil Takiar commented on IMPALA-8577: -- By default, it is only enabled if {{-Dorg.wildfly.openssl.path=[openssl-path]}} is added to {{HADOOP_OPTS}} otherwise it has no way of knowing where OpenSSL is located on the machine and falls back to the JSSE implementation. Users can set {{fs.s3a.ssl.channel.mode}} to {{Default_JSSE}} as well to force S3A to use JSSE instead of OpenSSL. I agree w.r.t the concerns about stability of wildfly SSL + AWS SDK, we could just set the default to {{Default_JSEE}} and mark the OpenSSL support as experimental / unstable. > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853293#comment-16853293 ] Steve Loughran commented on IMPALA-8577: s3a calls abort() on the AWS input stream when it needs to do a big seek; the AWS SDK then tries to close the TCP stream without flushing it, reading to the end etc. If somehow that aborted stream was returned to the connection pool, I can see bad things happening. I am starting to worry about the stability of the wildfly SSL +AWS SDK setup. Is it always automatic, or do we have a way of forcing use of the JDK version. I wan't to be able to field support calls with "try setting X" rather than "try removing JAR Y from all machines in your cluster" > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852607#comment-16852607 ] Sahil Takiar commented on IMPALA-8577: -- Running against master seems to have fixed double-free error, but now I'm hitting another error: {{org.wildfly.openssl [ERROR] error:140C30F0:SSL routines:SSL_set_session:unable to find ssl method}} although I've only ever seen this once. This looks like a bug in the session re-use handling in wildfly-openssl, perhaps similar to [https://github.com/wildfly/wildfly-openssl/pull/38] I don't have a full stack-trace, but the fact its failing in {{SSL_set_session}} makes me think the failure is caused by {{OpenSSLClientSessionContext#tryAttachClientSideSession}}. I've having trouble finding out what is producing the actual error message, but [https://markmail.org/thread/rf6kfqr5x63okkbl#query:+page:1+mid:7cqds6d57ob3dsxh+state:results] suggests that invalid parameters are being passed to {{SSL_set_session}}. > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852039#comment-16852039 ] Sahil Takiar commented on IMPALA-8577: -- I looped TPC-DS with {{fs.s3a.connection.maximum}} = 1 overnight (it didn't get through all of TPC-DS because its so slow), but I couldn't reproduce the issue (although it gets much farther in TPC-DS that my other runs that reproduce this error). Going to try with wildfly-openssl master now, with the original value of {{fs.s3a.connection.maximum}} (1500). > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851434#comment-16851434 ] Sahil Takiar commented on IMPALA-8577: -- hmm its suspicious this hasn't happened for ABFS, which makes me think perhaps there is a bug in the AWS SDK that is causing this. However, [https://github.com/wildfly/wildfly-openssl/issues/36] looks pretty suspicious (it reports a double-free error under high concurrency) and it was fixed in [https://github.com/wildfly/wildfly-openssl/pull/38] which has not made its way into a release yet. Going to try running against wildfly-openssl master and see what happens. Trying to confirm if this is concurrency related as well (playing around with the value of {{fs.s3a.connection.maximum}}). > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851329#comment-16851329 ] Da Zhou commented on IMPALA-8577: - no I didn't observed this issue on ABFS side. The only issue I found so far is that Wildfly-OpenSSL currently doesn't support Server Name Indication(SNI), but this would be fixed soon: [https://github.com/wildfly/wildfly-openssl/issues/59] > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851058#comment-16851058 ] Sahil Takiar commented on IMPALA-8577: -- I can actually re-produce this pretty easily. Just a single threaded TPC-DS execution on an ASAN build on the first iteration. I'm going to play around with netty-tcnative and see if it produces the error as well. It's possible there could be bug in the AWS SDK that is causing this; if that is the case then the crash should re-produce with netty-tcnative as well. > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850178#comment-16850178 ] Sahil Takiar commented on IMPALA-8577: -- Looking around, it seems like [netty-tcnative|https://netty.io/wiki/forked-tomcat-native.html] is another popular Java JSSE wrapper around OpenSSL (it uses the SSLContext API - see [this example|https://netty.io/wiki/forked-tomcat-native.html#wiki-h2-4]). It seems gRPC and Cassandra both support using netty-tcnative as well: * [https://docs.datastax.com/en/developer/java-driver/3.3/manual/ssl/#netty] * [https://github.com/grpc/grpc-java/blob/master/SECURITY.md] > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850110#comment-16850110 ] Sahil Takiar commented on IMPALA-8577: -- [~DanielZhou], [~vishwajeet.dusane] wondering if you have seen any similar issues when using Wildfly-OpenSSL in ABFS? Specifically, we are seeing crashes in Impala when enabling Wildfly-OpenSSL in S3A. The crash seems to be due to memory-corruption (a double free on the same block of memory). I haven't been able to track down the root cause of the issue, but I'm able to re-produce it in Impala while running against an [ASAN|https://en.wikipedia.org/wiki/AddressSanitizer] build. > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847942#comment-16847942 ] Sahil Takiar commented on IMPALA-8577: -- I can re-produce a similar error locally during an ASAN build (unfortunately the Java stack is missing): {code:java} ==61922==ERROR: AddressSanitizer: attempting double-free on 0x613000561280 in thread T179: #0 0x17e7c30 in free /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_m alloc_linux.cc:47 #1 0x7f58a642a0cc in CRYPTO_free (/lib/x86_64-linux-gnu/libcrypto.so.1.0.0+0x630cc) #2 0x7f58a684f02e (/lib/x86_64-linux-gnu/libssl.so.1.0.0+0x4302e) #3 0x7f58a6827add (/lib/x86_64-linux-gnu/libssl.so.1.0.0+0x1badd) #4 0x7f58a682d55e (/lib/x86_64-linux-gnu/libssl.so.1.0.0+0x2155e) #5 0x7f542977aab1 in Java_org_wildfly_openssl_SSLImpl_doHandshake0 (/tmp/tmp-6414427375012512006openssl/libwfssl.so+0x9ab1) #6 0x7f589038ad90 () 0x613000561280 is located 0 bytes inside of 352-byte region [0x613000561280,0x6130005613e0) freed by thread T187 here: ==61922==AddressSanitizer CHECK failed: /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_descriptions.cc:176 "((id)) != (0)" (0x0, 0x0) #0 0x17f2a2f in __asan::AsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_rtl.cc:69 #1 0x180e705 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/sanitizer_common/sanitizer_termination.cc:79 #2 0x1723be4 in GetStackTraceFromId /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_descriptions.cc:176 #3 0x1723be4 in __asan::HeapAddressDescription::Print() const /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_descriptions.cc:398 #4 0x1725227 in __asan::ErrorDoubleFree::Print() /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_errors.cc:117 #5 0x17eed0b in __asan::ErrorDescription::Print() /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_errors.h:374 #6 0x17eed0b in ~ScopedInErrorReport /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_report.cc:177 #7 0x17eed0b in __asan::ReportDoubleFree(unsigned long, __sanitizer::BufferedStackTrace*) /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_report.cc:279 #8 0x171fe3a in __asan::Allocator::ReportInvalidFree(void*, unsigned char, __sanitizer::BufferedStackTrace*) /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_allocator.cc:652 #9 0x171fe3a in __asan::Allocator::AtomicallySetQuarantineFlagIfAllocated(__asan::AsanChunk*, void*, __sanitizer::BufferedStackTrace*) /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_allocator.cc:523 #10 0x171fe3a in __asan::Allocator::Deallocate(void*, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType) /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_allocator.cc:597 #11 0x171fe3a in __asan::asan_free(void*, __sanitizer::BufferedStackTrace*, __asan::AllocType) /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-ub1604-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_allocator.cc:805 #12 0x17e7c0c in free
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[ https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846294#comment-16846294 ] Sahil Takiar commented on IMPALA-8577: -- Haven't found a root cause, but here is what I have found out so far: * My current theory is that there is a bug somewhere in Wildfly-OpenSSL that is corrupting memory; maybe a double free ** If we could reproduce this in an ASAN build that would point us to the error; maybe we can try running exhaustive tests for Impala-S3 a few times to see if this re-produces * Based on my understanding of the Wildfly-OpenSSL code, the crash occurred after S3 sent a "close notify" signal to the Wildfly client, which caused the Wildfly code to close the underlying BIOs (OpenSSL's file / socket abstraction) ** I think the stack might be a little incomplete, it should be freeBIO0 --> BIO_free --> OPENSSL_free --> CRYPTO_free (I think) * So maybe there is a bug where the Wildfly code double frees the BIO associated with the current SSL connection I'm going to try to run some ASAN builds and see if this re-produces. > Crash during OpenSSLSocket.read > --- > > Key: IMPALA-8577 > URL: https://issues.apache.org/jira/browse/IMPALA-8577 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: David Rorke >Assignee: Sahil Takiar >Priority: Major > Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp, > hs_err_pid6313.log, > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313, > > impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313 > > > Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from > the stack trace (hs_err log file attached with more complete stack): > {noformat} > Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free > space=8180k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > C [impalad+0x2528a33] > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, > unsigned long, int)+0x133 > C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f > C [impalad+0x266468a] operator delete(void*)+0x32a > C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d > J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ > 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9] > J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ > 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c] > J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ > 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c] > J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ > 0x7f3d4de419cc [0x7f3d4de417c0+0x20c] > J 2994 C2 > org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; > (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134] > J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ > 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0] > J 5090 C2 > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I > (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0] > J 5846 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I > (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384] > J 5845 C1 > com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 > bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c] > {noformat} > The crash may not be easy to reproduce. I've run this test multiple times > and only crashed once. I have a core file if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org